Hide Notes 

Slide 1: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 DRAFT Quality Assurance Surveillance Plan (QASP) For Installation Services Defense Logistics Agency (DLA) Defense Distribution Center (DDC) Defense Distribution Depot San Joaquin, California (DDJC) dd mmm 2006 1
Slide 2: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Quality Assurance Surveillance Plan (QASP) Section A. 1.1 Introduction Overview and Organization As defined in Federal Acquisition Regulation (FAR) 37.601, the Defense Distribution Depot San Joaquin, California (DDJC) contract for Installation Services was issued as a performance-based solicitation. A performance-based contract is intended to ensure that the required performance quality levels are achieved and that total payment is related to the degree that services performed or outcomes achieved meet contract standards. In order to fulfill the requirements for performance-based contracting, the contract must use measurable performance standards (i.e., in terms of quality, timeliness, quantity, etc.) and quality assurance surveillance plans. This Quality Assurance Surveillance Plan (QASP), which complies with the requirements of the FAR, serves as a guide in monitoring the Service Provider’s performance in accordance with Request for Proposal (RFP), solicitation number SP0700-05-R-7018. Under the provisions of FAR 37.602-2, agencies are required to develop quality assurance surveillance plans when acquiring services through performance-based contracts. The quality assurance surveillance plans must recognize the responsibility of the Service Provider to carry out its Quality Control/Customer Satisfaction Plan (QC/CSP) obligations and shall contain measurable inspection and acceptance criteria corresponding to the performance standards contained in the performance work statement. The quality assurance surveillance plans must focus on the level of performance required rather than the methodology used by the Service Provider to achieve that level of performance. Contract performance incentives, whether positive or negative or both, which are incorporated to the maximum extent possible into performance-based contracts to encourage Service Providers to increase efficiency and maximize performance, must correspond to the specific performance standards in the quality assurance surveillance plan and must be capable of objective measurement. The FAR, including Part 46, Quality Assurance, “prescribes policies and procedures to ensure that supplies and services acquired under Government contract conform to the contract’s quality and quantity requirements. Included are inspection, acceptance, warranty, and other measures associated with quality requirements.” This QASP, which complies with the requirements of FAR Part 46, is organized into five major conceptual groupings: Introduction, Surveillance Methods, Surveillance Activities for Specific Tasks, Documentation Requirements, and Helps and Aids. The content outline below identifies further subdivisions of these sections. This outline also serves as the Table of Contents for the QASP. The various headings and page references are hyperlinked to detailed topical discussions in the QASP. 2
Slide 3: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Quality Assurance Surveillance Plan Table of Contents Quality Assurance Surveillance Plan (QASP)......................................................................................2 As part of Acquisition Planning, which occurs prior to the release of a solicitation, identification of quality assurance requirements are among the many logistical factors considered. After contract award, the Government uses the QASP as the document to enforce the inspection and acceptance clauses of the RFP. Recognizing that the Service Provider’s Quality Control/Customer Satisfaction Plan influences the final QASP, the QASP may require revision after solicitation award to reflect the Service Provider’s QC/CSP strengths and weaknesses. Ideally, the Government shares the QASP, along with any updates, with the Service Provider’s Facilities Manager during Phase-In and throughout the life of the contract1 to help achieve the best possible Service Provider performance. 1.2 Types of Contract Quality Requirements The FAR (46.202) identifies four general categories of contract quality requirements, depending on the extent of quality assurance needed by the Government for the acquisition involved: contracts for commercial items, government reliance on inspection by Service Provider, standard inspection requirements, and higher-level contract quality requirements. The primary category used in the contract is standard inspection requirements which: • • • 1.3 Require the Service Provider to provide and maintain an inspection system that is acceptable to the Government; Give the Government the right to make inspections and tests while work is in progress; and Require the Service Provider to keep complete and make available to the Government, records of its inspection work. Considerations in Developing the QASP This QASP specifically corresponds to the contract, particularly the Performance Work Statement (PWS) contained in Section C of the contract. As mentioned previously, it also complies with the policies and procedures of FAR Part 46, Quality Assurance. Also incorporated is the requirement to perform the inspection and tests in such a fashion as to not unduly delay the work performed by the Service Provider (see FAR 8.406-2). The PWS document identifies the standards for performance for accomplishing the work requirements contained in the contract. The QASP provides the surveillance plan necessary to ensure acceptable performance. In addition to the Installation Services, the following identify additional factors in QASP development: • 1 Reduced Defense Distribution Center (DDC) budget In the context of this QASP, contract and MEO Letter of Obligation (for MEO decisions) are interchangeable. 3
Slide 4: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • Productivity improvements Limited resources in terms of time and personnel are available for surveillance The QASP’s main objective remains to evaluate the Service Provider’s performance across all requirements relative to performance standards; therefore, the QASP follows the same category groupings contained in the contract. These categories help identify the level of effort that leads to effective provision of support services. The primary interest of the Government is the acceptability of the service or final product provided by the Service Provider, rather than the detailed operational procedures or processes utilized to provide the service. However, the Government’s review of the Service Provider’s operational processes and procedures provides insight and understanding into the Service Provider’s ability to provide acceptable services and products. When quality issues arise, the Government may review or audit the Service Provider’s processes and procedures to detect potential causes for the inability to provide acceptable services and products. FAR 9.104-1 specifies that in order for a prospective Service Provider to be determined responsible2, the Service Provider must “Have the necessary organization ...and technical skills, or the ability to obtain them (including, as appropriate, such elements as... quality assurance measures and safety programs applicable to materials to be produced or services to be performed...” Additionally, one of the contract requirements calls for the Service Provider to provide and maintain a QC/CSP that addresses methods for meeting performance standards and requirements and complying with applicable regulations as detailed in the contract. The selected Service Provider’s QC/CSP, submitted as an element of its proposal, became part of the contract upon award. Since the QC/CSP and any revisions are also subject to the inspection and acceptance clauses of the contract (see Section E of the contract), the Government uses the QASP to ensure the effectiveness of the Service Provider’s QC/CSP. The Government should share the final version of the QASP, along with any updates, with the Service Provider’s Contract Manager during Phase-In and full performance to help achieve the best possible Service Provider performance. 1.4 Timing of QASP Development According to FAR Subpart 46.4, Government Contract Quality Assurance, quality assurance surveillance plans should be prepared in conjunction with the preparation of the Performance Work Statement (PWS). The plans should specify— (1) All work requiring surveillance; and (2) The method of surveillance. The FAR further specifies that the contract shall designate the place or places where the Government reserves the right to perform quality assurance. If the contract is for services performed at a destination, quality assurance inspections shall be performed at the destination locations. Additionally, all Government inspection shall be performed by or under the direct supervision of Government personnel and Government inspection shall be documented on an inspection or receiving report form under agency procedures. A responsible prospective Service Provider must meet the standards of FAR 9.104. Only a responsible prospective Service Provider is considered for a contract award. 2 4
Slide 5: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Since the draft QASP was developed prior to award, many variables were unknown at the time of initial development, including: • • • • • • Critical changes in performance requirements Changes to governing directives and laws Changes to the DDC’s approach to conducting assessments of Service Provider performance Major reorganizations (e.g. of the DLA, DDC or DDJC) that could affect the QASP The Selected Service Provider’s Quality Control procedures (e.g., QC/CSP) The Selected Service Provider’s performance strengths and weaknesses Therefore, the QASP may require revision after contract award to reflect some of the variables listed above, especially those that address the Service Provider’s QC/CSP and known strengths and weaknesses. QASP modification or revision may also be required to reflect changes due to process improvements, quality issues, or inclusion of new surveillance techniques. 1.4.1 Responsibilities for Quality Assurance Responsibilities for Contract Quality Assurance are well-defined in FAR Part 46 for the Contracting Office, the Administrative Contracting Office, the Requiring Activity and the Service Provider. Contracting Office • • • • • Receiving from the activity responsible for technical requirements, any specifications for inspection, testing, and other contract quality requirements essential to ensure the integrity of the services. Including in solicitations and contracts the appropriate requirements for the Service Provider’s control of quality for the services to be acquired Issuing any necessary instructions to the cognizant contract administration office and acting on recommendations submitted by that office Verifying that the Service Provider fulfills the contract quality requirements Ensuring that non-conformances are identified, and establishing the significance of a nonconformance when considering the acceptability of services which do not meet contract requirements. Administrative Contracting Office • • Develops and applies efficient procedures for performing government quality assurance actions under the contract in accordance with the written direction of the contracting office Performs all actions necessary to verify whether the services conform to contract quality requirements 5
Slide 6: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • Maintain, as part of the performance records of the contract, suitable records reflecting – ♦ The nature of Government contract quality assurance actions, including, when appropriate, the number of observations made and the number and types of nonconformance; and ♦ Decisions regarding the acceptability of the products, the processes, and the requirements as well as action to correct nonconforming items. Implement any specific written instructions from the contracting office Report to the contracting office any non-conformances observed in design or technical requirements, including contract quality requirements; and, Recommend any changes necessary to the contract, specifications, instructions, or other requirements that will provide more effective operations or eliminate unnecessary costs. • • • Requiring Activity • • Develops and provides to the Contracting Office, specifications for inspection, testing and other contract quality requirements essential to ensure the integrity of the services Prescribes contract quality requirements, such as inspection and testing requirements, or, for service contracts, a quality assurance surveillance plan Service Provider The Service Provider is responsible for carrying out its responsibilities under the contract by – • • • • Controlling the quality of services; Tendering to the Government for acceptance only those services that conform to contract requirements; Ensuring that vendors or suppliers of raw materials, parts, components, subassemblies, etc., have an acceptable Quality Control/Customer Satisfaction Plan system; and Maintaining substantiating evidence, when required by the contract, that the services conform to contract quality requirements, and furnishing such information to the Government as required. The control of quality by the Service Provider may relate to, but is not limited to – • • • • Manufacturing processes to ensure that the product is produced to, and meets, the contract’s technical requirements; Drawings, specifications, and engineering changes, to ensure that manufacturing methods and operations meet the contract’s technical requirements; Testing and examination, to ensure that practices and equipment provide the means for optimum evaluation of the characteristics subject to inspection; Reliability and maintainability assessment (life, endurance and continued readiness); 6
Slide 7: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • • Fabrication and delivery of products, to ensure that only conforming products are tendered to the Government; Technical documentation, including drawings, specifications, handbooks, manuals and other technical publications Procedures and processes for services to ensure that services meet contract performance requirements. The Service Provider is responsible for performing all inspections and test required by the contract except those specifically reserved for performance by the Government. Government The Government’s QASP responsibilities remain simple and straightforward: • • • • 1.5 Develop the QASP as a viable, reliable, and effective tool for monitoring and evaluating Service Provider compliance to contract standards and requirements Train all appropriate personnel in the purpose, use, and implementation of the QASP Monitor and review the QASP for effectiveness Update the QASP as needed to maintain viability and effectiveness QASP Principles The acceptance and inspection clauses in the RFP (Section E) allow the Government to implement quality assurance (QA) procedures. Other contract clauses require the Service Provider to implement a QC/ CSP, resulting in the Service Provider remaining responsible for Quality Control/Customer Satisfaction Plan assessments of services produced. The Government remains responsible for QA, surveillance, monitoring, and evaluation of the Service Provider’s quality performance. This concept is well defined in FAR 46.101, which says Government contract quality assurance means the various functions, including inspection, performed by the Government to determine whether a Service Provider has fulfilled the contract obligations pertaining to quality and quantity (emphasis added). The QASP represents the Government's acceptance and inspection program for contract requirements. Agencies must ensure that Government quality assurance is conducted before product or service acceptance by or under the direction of Government personnel. The QASP documents this program in order to provide a measure of the quality and timeliness of products and services purchased from the Service Provider. The Government, as recipient of the Service Provider’s products and services, retains the responsibility for developing and implementing methods for QA. Implementation of the QASP assists in providing validation that the quantity and quality of products and services received comply with contract requirements. The QASP focuses on corroborating the quality and timeliness of the products and services received from the Service Provider and the fulfillment of contract requirements including those mandatory requirements commonly called “Service Provider Shalls.” The QASP details the following: 7
Slide 8: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • • • Methods for surveillance of each contract requirement Methods for identifying categories of performance nonconformance, as defined in FAR Part 46 including, critical, major and minor nonconformance. Evaluation procedures to be used for each surveillance method Approaches for QASP implementation This QASP recommends specific Quality Assurance (QA) approaches for each required service. The approaches provide step-by-step procedures explaining performance and documentation requirements of the evaluation processes, analysis of evaluation results, and determination of satisfactory or unsatisfactory Service Provider performance. 1.6 QASP Implementation and Execution Successful implementation of the QASP is based upon careful planning and targeted use of the following: • • • • • Surveillance Planning and Scheduling Surveillance performance, ensuring documentation Data analysis of surveillance results Consistent performance of QASP requirements Frequent QASP updates based on contract modifications, surveillance results or Service Provider improvements The following four objectives are the cornerstones for the Government’s daily surveillance activities and remain critical to QASP implementation: • • • • Monitor the Service Provider compliance status to established Acceptable Performance Level (APL) and non-APL requirements Verify the Service Provider’s compliance with the Service Provider’s QC/CSP Document surveillance observations and findings Analyze and report surveillance results to the Service Provider Surveillance Planning and Scheduling 1.6.1 An annual Surveillance Plan, which identifies the surveillance activities, frequencies, priorities and points of contact (POCs) for monitoring the Service Provider’s performance requirements, documents the Government’s inspection objectives for the year. The annual Surveillance Plan is forwarded, in turn, to the 8
Slide 9: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Contracting Officer’s Representative3 (COR), and the Contracting Officer (KO)/Administrative Contracting Officer (ACO)4, for approval. Based on the approved annual surveillance plan requirements, the CGA develops a monthly surveillance schedule to identify required monthly surveillance events and frequencies. Government personnel update surveillance records to clearly identify the surveillance activities and results for the month. The CGA completes development of the monthly schedule by the last workday of the preceding month. The COR reviews the monthly surveillance schedule and forwards it to the DDJC Commander or Facilities Director, for approval. The CGA modifies the monthly surveillance schedules as required to achieve targeted use objectives. The CGA incorporates time to validate user complaint items into the surveillance schedule to the extent possible. Key among the CGA’s responsibilities are scheduling surveillance and tracking, analyzing and reporting surveillance results. During all periods of Service Provider performance, the CGA retains a copy of all surveillance schedules, User Complaint Records (UCR), and Service Provider Surveillance Report. The CGA forwards these records to the KO at the end of each performance period for inclusion in the contract file. 1.6.2 QASP Documentation Successful QASP implementation produces the following documentation: • • • • • • • Forms Logs Reports Photographs Letters, Notes, and/or E-mails Charts and Graphs Evaluations Forms used for documentation (e.g., Service Provider Surveillance Reports and User Complaint Records [UCR]) are single-page documents designed to promotes continuous and consistent documentation. To some degree, surveillance of Service Provider quality and performance requirements remains subjective in nature. Therefore, continuous and consistent documentation allows the evaluator to build more objectivity into the process. Documentation of Service Provider performance provides a legal basis for the Government to take action and make decisions. Informal or anecdotal evidence does not provide a legal basis to reward the Service Provider or to initiate corrective actions. It is equally important to document both satisfactory and unsatisfactory performance. Documentation of satisfactory performance assists in validating proper implementation and execution of contract requirements. Documenting satisfactory performance assists in In the context of this QASP, COR and COR-equivalent are interchangeable. FAR Sub Part 2.1 identifies differences between the Contracting Officer (KO) and an Administrative Contracting Officer (ACO); however, for purposes of this QASP, ACO and KO are interchangeable. 3 4 9
Slide 10: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 identifying Service Provider approaches that work which provide ideas for improvement. The vast majority of surveillance generally results in documenting satisfactory performance. Documentation does not require complex writing; rather, the use of simple and clear sentences is preferred. Accuracy and completeness remain the keys to successful documentation. At a minimum, the documentation should allow for complete understanding based solely on the information contained in the documentation. Following the “Rules of Documentation” listed below is also important: • • • • • Use only black or blue ink Fill in all information blocks and blanks If a particular informational block does not apply, insert “N/A” Never erase or use correction fluid/tape on a document When errors occur  Draw a single line through the error  Initial and date the line-through  Insert the correct information above or below the line-through Always sign and date documents with the full date (i.e., month, day, year) QASP Modifications • 1.6.3 The Government may unilaterally modify the QASP. Such modifications are not subject to the modification clauses in the contract and are not grounds for increasing the cost of performance. While modifications refer to changes in the QASP requirements, as reflected in contract requirements, modifications do not refer to monthly adjustments in surveillance schedules by individual Government personnel or other representatives. Due to unique responsibilities and ideal positioning, Government Surveillance Specialists are ideally suited to recommend QASP modifications. Government personnel should assemble and present all recommended QASP modifications to the CGA and COR for forwarding to the KO for approval. In an effort to achieve optimum Service Provider performance, the Government (i.e., typically via the COR) may inform and provide the Service Provider with QASP modifications for information purposes only. The CGA distributes all approved QASP modifications to Government personnel. This distribution includes instructions for purging deleted pages and adding revised pages to all QASP manuals. 1.6.4 QASP Execution The QASP provides the direction and approaches for execution of the surveillance and monitoring activities of the Government. During the initial months of performance under the contract, the Government closely follows the QASP to encourage the Service Provider to adhere to the requirements of the contract. As all parties become familiar with the requirements, operational environment and the processes and 10
Slide 11: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 procedures necessary to ensure successful performance, surveillance and monitoring activities typically alter to meet the needs of the changing DDJC environment. In addition, the Government should target surveillance events toward the most critical problems existing at any given time. At the same time, the Government must consistently verify that the Service Provider continues to execute all contract requirements and APLs. Section B. 1.1 Surveillance Methods Background The Government applies, individually or in combination, the following surveillance/monitoring methods in order to verify Service Provider performance of the contract requirements. These methods apply to surveillance/monitoring of functional responsibilities. Government members record observations, using a Service Provider Surveillance Report, during the use of any of the following surveillance methods. 1.2 Surveillance Outcomes Results of surveillance activities or events document either satisfactory or non-conforming (unsatisfactory) Service Provider performance of contract requirements. Services shall ordinarily not be accepted before completion of Government contract quality assurance actions (see FAR Part 46.5). 1.2.1 Satisfactory Outcomes Satisfactory outcomes meet the intent of contract requirements. Most Service Provider outcomes are satisfactory. As with innocence in a courtroom, the Service Provider’s outcomes and processes are presumed to be satisfactory unless proven otherwise. 1.2.2 Nonconforming Outcomes Part 46, Quality Assurance, of the Federal Acquisition regulation defines three types of nonconforming services: Critical Nonconformance: A nonconformance that is likely to result in hazardous or unsafe conditions for individuals using, maintaining, or depending upon the services; or is likely to prevent performance of a vital agency mission. Major Nonconformance: A nonconformance, other than critical, that is likely to result in failure of the services, or to materially reduce the usability of the services for their intended purpose. Minor Nonconformance: A nonconformance that is not likely to materially reduce the usability of the services for their intended purpose, or is a departure from established standards having little bearing on the effective use or operation of the services. When nonconforming services are identified, the contracting officer ordinarily must give the Service Provider an opportunity to correct or replace the nonconforming services when this can be accomplished within the required delivery schedule. Unless the contract specifies otherwise, correction or replacement must be without additional cost to the Government. 11
Slide 12: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 If the nonconformance is minor, the cognizant contract administration office may make the determination to accept or reject, except where this authority is withheld by the contracting office of the contracting activity. To assist in making this determination, the contract administration office may establish a joint Service Provider-contract administrative office review group. Acceptance of services with critical or major non-conformances is outside the scope of the review group. The contracting officer must discourage the repeated tender of nonconforming services, including those with only minor non-conformances, by appropriate action, such as rejection and documenting the Service Provider’s performance record. When services are accepted with critical or major non-conformances, the contracting officer must modify the contract to provide for an equitable price reduction or other consideration. For services, the contracting officer can consider identifying the value of the individual work requirements or tasks that may be subject to price or fee reduction. This value may be used to determine an equitable adjustment for nonconforming services. However, when services involving minor non-conformances are accepted, the contract need not be modified unless it appears that the savings to the Service Provider in fabricating the nonconforming services will exceed the cost to the Government of processing the modification. Notices of rejection must include the reasons for rejection and be furnished promptly to the Service Provider. Promptness in giving this notice is essential because, if timely nature of rejection is not furnished, acceptance may, in certain cases, be implied as a matter of law. The notice must be in writing if – • • • The services have been rejected at a place other than the Service Provider’s plant; The Service Provider persists in offering nonconforming services for acceptance; or Delivery or performance was late without excusable cause. Isolation of Nonconforming (Discrepant) Service If nonconforming (discrepant or defective) services are detected during surveillance (QC/CSP or QA), but the quantities of nonconformance are not sufficient to warrant rejecting the lot, return the acceptable samples to the lot. Remove and isolate the defective samples from the lot and complete the required documentation (e.g., XXXX). When using a statistically significant number of samples to evaluate a population (lot) and the inspected samples indicate that the lot fails to comply with the expected quality levels, isolate the entire lot (e.g., XXXX, etc.). 1.3 Direct Observation/Random Visual Surveillance/Monitoring The act of observation involves examining or watching attentively. The Government may use direct observation of services and products to ensure requirement implementation. The evaluator may use direct observation periodically or during 100% surveillance/monitoring. This surveillance/monitoring involves the examination of services to determine whether services conform to contract requirements. 12
Slide 13: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.4 Periodic Surveillance/Monitoring The periodic surveillance/monitoring method uses a comprehensive evaluation of selected outputs. This method measures quality for interim and final outputs. The Surveillance Specialists may perform scheduled or unscheduled inspections (e.g., daily/weekly/monthly/etc.). 1.5 One Hundred Percent (100%) Inspection One hundred percent inspection evaluates all outputs of a particular requirement and applies mostly to a small quantity of products or services, but may also be used to evaluate important or critical products and services. The Surveillance Specialist should carefully consider the use of 100% inspection due to the demands on resources, such as the Government’s limited work force and time constraints. Note: One hundred percent inspections typically result in only 85% accuracy. Therefore, the use of 100% inspection requires careful consideration. 1.6 Validated User/Customer Complaints User/customer complaint reports rely on the user or the customer to identify deficiencies. If the Service Provider cannot satisfy a customer with a proposed corrective action, the Surveillance Specialists investigate and validate the customer complaint. Even the best surveillance plan possesses limits in checking all aspects of the Service Provider performance. The Surveillance Specialists carefully plan the manner of obtaining and documenting unresolved user complaints. Validated complaints constitute a surveillance method based on user awareness. Customers familiar with contract requirements and unable to receive a satisfactory response from the Service Provider may notify or request assistance from the Government. Upon notification, the Government member fills out a User Complaint Record (UCR) and conducts surveillance/monitoring to validate or invalidate the complaint. The Government may not use customer complaints to satisfy a QASP requirement to accomplish random observations or 100% inspection requirements. However, the Government may use a customer complaint as further evidence of unsatisfactory performance, especially if periodic or 100% inspections show unsatisfactory service in specific areas. The Government may also use customer complaints to decide if the unsatisfactory performance requires additional surveillance. 1.7 User Survey A user survey combines elements of validated user complaints and random sampling. The Government may conduct a random survey in order to solicit user satisfaction. This process is appropriate for high quantity activities that have historically been satisfactory. This method may also generate periodic and 100% inspections requirements. The Government receives the survey responses and then performs an initial review to identify negative responses. The Government reviews Service Provider-provided reports, as required by the Service Provider’s QC/CSP, to help identify trends and general issues. Because of this review and validation effort, the Government then documents survey results on a Service Provider Surveillance Report. 13
Slide 14: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.8 Random Sampling Random sampling evaluates the outputs of contract requirements by randomly selecting and inspecting a statistically significant sample based on the lot size of the material. Surveillance Specialists use this approach for large quantity, repetitive activities with objective and measurable quality characteristics. 1.8.1 a) Steps to Perform Random Sampling: Determine what constitutes the lot or population. A lot is, “Any homogeneous group of items or services received or produced during a single event or production run.” A “lot” can be a single service order completed, equipment work order completed, or the total number of equipment work orders completed per month. Basing the lot size on a smaller quantity of material (such as a service order completed) allows the Surveillance Specialist to accept or reject the material by the smaller quantity (i.e., service order completed) rather than a larger quantity (i.e., total completed equipment work orders). Find the sample size and the “accept” and “reject” numbers or limits. The sample size and the “accept” and “reject” numbers or limits are found by referencing the Random Sampling Plan (see critical, major and minor AQL columns). b) Randomly select the samples from the lot. After determining, the sample size and accept/reject numbers, randomly select the samples from the lot. 1.8.2 Reduced Surveillance/Monitoring When a Service Provider's quality-control program works, the result equals good performance. If the results of the surveillance events show consistently satisfactory performance, the number of random-sampling surveillance may decrease. In coordination with the COR (or equivalent), DDJC Commander and KO (or equivalent) develops a schedule, which incorporates reduced surveillance/monitoring when all of the following conditions occur: • • The preceding four lots, e.g., the last four months, were acceptable The number of non-conformances in each of the preceding four lots equals less than one-half of the acceptable number; e.g., with an APL of 10% and a sample size of 20, the acceptance number equals five, and then only two or less non-conformances become acceptable in each of the last four lots. A normal sample size was used. The Contracting Officer (KO) agrees with the use of reduced inspections. • • Reduced inspection decreases the sample size used to evaluate Service Provider performance. In addition, the acceptance and rejection numbers change for the given APL. To determine the sample size and the “accept”/“reject” numbers for reduced inspection, refer to the Random Sampling Scheme for Reduced Inspection (page 61). 14
Slide 15: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Note: If a gap between the acceptance and rejection numbers exists, (e.g., with a sample size of 20 and an APL of 10%) accept equals five and reject equals eight. This means that the Surveillance Specialists reject a lot when eight or more non-conformances occur. However, non-conformances greater than five justifies returning to normal inspections, i.e., returning to the sample size and acceptance and rejection numbers used in the original sampling guide. When performing reduced inspections, the KO (or equivalent) adjusts the surveillance schedule to return to normal inspection the next surveillance period under the following conditions: • • When the number of nonconforming items exceeds the acceptance number under reduced sampling When the KO deems it necessary to return to normal inspection If, during the first month after normal-sampling resumption, the number of nonconforming items becomes less than 50% of the reject threshold, the KO (or equivalent) shall adjust the schedule to return to reduced inspections the next month. If nonconforming items totaling more than 50% of the reject level are identified, the CGA develops a surveillance schedule, which implements normal sampling, until four months with fewer than 50% of the reject level of nonconforming items occurs. 1.9 Periodic Sampling Periodic sampling is a variation of random sampling. Only when the Government suspects that a deficiency exists should the Government perform periodic sampling. The Government may identify this suspected deficiency through analysis of IFS or EMACS information, for example, and performing periodic sampling remains a good follow-up to IFS or EMACS analysis. Customer complaints for a specific area could become another trigger for periodic sampling. The sampling results apply only for the specific work inspected. Since a random sample does not occur, the Government cannot use periodic sampling to draw conclusions about all the activities performed by the Service Provider. 1.10 Eight Step Surveillance Process The list below identifies an eight-step process for performing surveillance: a) b) c) d) e) f) g) h) Identify the APL/Non-APL Criterion Review all relevant documentation Fill out the Surveillance Report Conduct surveillance Document observations Analyze data Report findings File documents 15
Slide 16: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Incorporating these eight steps into the Government’s surveillance methods provides a direct contribution to the success of the Government’s performance assessment program. 1.11 Surveillance Method Selection Another key to successful surveillance and monitoring rests with the ability of the Surveillance Specialist to select the proper surveillance method appropriate for the observed activity. The Surveillance Method Selection Tool, provides a systematic process for selection of appropriate surveillance methods. Seven examples are also provided. These examples use the Surveillance Method Selection Tool and illustrate the use of the tool and the Eight-step Surveillance Process. Each example uses a different scenario and surveillance method. The following list provides definitions key to the successful use of the Surveillance Method Selection Tool: As Directed: Unplanned or unscheduled instructions for conducting surveillance or monitoring activities. Chance Discovery: A fortuitous detection of a discrepancy while engaged in another activity. Critical Process: A course of action considered vital. Customer/User Complaint: An issue of noncompliance raised and reported by a customer (example: damaged material arriving at the customer’s location caused by insufficient packaging). Non-APL Requirement: A PWS requirement (e.g., Service Provider shall), which the service provider is required to comply with 100% of the time. Non-Critical Process: A course of action considered important, but not vital or hazardous to life or safety; usually a non-APL. Not at DDJC: The items under surveillance are not at DDJC. Acceptable Performance Level (APL) Requirement: The minimum performance acceptable for each designated requirement before the government considers performance unsatisfactory. 16
Slide 17: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan At DDJC APL 11/25/11 Potential Surveillance Methods Random Sampling, Periodic Sampling, Periodic Monitoring, Direct Observation, Random Visual Surveillance/Monitoring, 100% Inspection Not at DDJC Potential Surveillance Methods User Survey, Validated User/Customer Complaints, Direct Customer Contact Scheduled Surveillance NON APL Critical Process Potential Surveillance Methods 100% Inspection, Direct Observation Potential Surveillance Methods Non Critical Random Sampling, Periodic Sampling, Periodic Monitoring, Direct Observation, Random Visual Surveillance/Monitoring, 100% Inspection START HERE: GOAL Verify Service Provider Compliance with contract As Directed Potential Surveillance Methods 100% Inspection, Direct Observation, Random Visual Surveillance/Monitoring, Random Sampling, Periodic Sampling, Periodic Monitoring Suspected Deficiency Unscheduled Surveillance Customer User Complaint Chance Discovery Potential Surveillance Methods Direct Observation, IFS, EMACS, Validated User/Customer Complaints, Random Sampling, 100% Inspection Figure 0-1, Surveillance Method Selection Tool Scheduled Surveillance: Planned Government monitoring activities. Normally scheduled through the KO (or equivalent) and approved by the COR. Suspected Deficiency: Assumed or foreknowledge of nonconforming material (example: XXX). Unscheduled Surveillance: Unplanned monitoring activities usually arising from a crisis, immediate direction from leaders, or chance discoveries (example: XXX). 17
Slide 18: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.11.1 Surveillance Examples TBD (Develop Installation Services Samples using each of the surveillance methods) 1.12 Surveillance Preplanning Preplanning surveillance activities takes a little time and effort, but pays off big by providing a comprehensive approach for conducting surveillance activities past, present and future. Preplanning helps the Government focus on what needs to be done and when, thus making progress and goal attainment easier. Preplanning provides structure and direction to the Government’s efforts, which in-turn, help to eliminate the stress of day-to-day firefighting. Knowing what you have already done and what remains will only make the job easier. 1.12.1 Manual Surveillance Plan Steps to take when developing a surveillance plan are: • • • • • • • List the known APL activities requiring monitoring. List known customer complaints. List known problem areas detected by previous surveillance. List contract requirements not associated with an APL. List special projects or surveillance assignments from the commander or others. Prioritize the compiled list. Priorities change over time so the Government needs to be flexible. Create a word table with the following headings and columns (see Sample Grounds Maintenance and Repair Surveillance Plan Matrix): o o Surveillance Activity & Requirement (one column). Create a column for each day for a seven (7) day period or for the entire month (one column for each day). Identify the Time of Surveillance Enter the surveillance activities and requirements in the proper column. The order of entry is not important. Enter the time to conduct the surveillance under the selected date. Select time, date and frequency of surveillance based on the prioritized list already compiled. o o o • Determine priority by the placement and frequency within the week a particular activity needs monitored. 18
Slide 19: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • Frequency can be multiple times per day, on several different days, or a combination of multiple times per day and days per week. Grounds Maintenance and Repair Surveillance Plan Date: Surveillance Activity & Requirement Street sweeping services for paved roads, streets and parking lots • Observe: Once every two weeks year round weather permitting. After sweeping, pavements shall be free of dirt, debris, and foreign matter. Plant should die or become unhealthy due to Service Provider damage: • Check replacement of the damaged plant with a plant of same size and type, at no additional cost to the Government and within five working days of noticed damage. Improved Grounds: • Observe that the growth of grass height on improved grounds is maintained to have a well manicured look at all times at an average height of 2 ½ to 3 inches. Litter and Debris Removal: • Police high traffic building parking lots and improved grounds (Buildings 100, 16, 30 at Tracy; and 330 at Sharpe) once during each week. • Observe during the fall months that fallen leaves removed weekly from all affected areas. Semi Improved Grounds • Observe that vegetation maintained to a height of 10 inches or less to ensure plant health. Unimproved Grounds • Observe that vegetation maintained to a height of 14 inches or less Pest Management Services: • Check DDJC Pest Management Plan for implementation • Check all records of pesticide use accomplished using IPMIS • Receive the Pest Management Report, quarterly and annually [Date] Mon [Date] Tue [Date] Wed [Date] Thu [Date] Fri [Date] Sat [Date] Sun Time & Frequency of Surveillance 19
Slide 20: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Grounds Maintenance and Repair Surveillance Plan Date: Surveillance Activity & Requirement • Observe monthly scheduled inspections and treatments in facilities. • Check monthly IFS SO report [Date] Mon [Date] Tue [Date] Wed [Date] Thu [Date] Fri [Date] Sat [Date] Sun Time & Frequency of Surveillance Figure 0-2, Sample Grounds Maintenance and Repair Surveillance Plan Matrix Section C. Surveillance Activities for Specific Tasks The surveillance activities for specific tasks provided below provide insight into the approaches the Government should consider using during surveillance of these activities. 1.1 Management and Administration Management and Administration includes, but is not limited to, work reception, administration of work orders, weekly meetings, and customer coordination. Recurring requirements are the regular and recurring operations requirements such as the daily operation of the wastewater treatment plan, water distribution system and boilers, daily testing of water and grass mowing, and Preventive Maintenance (PM) on facilities and equipment. The Service Provider performs PM, which is the cyclic check, inspection and servicing of facilities and equipment as identified in the Technical Exhibits. PM is scheduled recurring work designed to maintain operational capability, increase asset life and reduce life cycle costs. The Service Provider performs and maintains a PM program IAW manufacturer’s specification during the warranty period and thereafter IAW the schedules provided in the TEs unless environmental conditions, safety concerns or publication changes necessitate a change. The Service Provider plans schedule and perform the work for Service Order (SO) in IFS. The Service Provider receives Emergency Priority 1 work from the Government either verbally (e.g., radio) or in writing. The Service Provider performs work in this category with an IFS SO to follow. The Service Provider begins work to Priority 1 SOs within 30 minutes after being notified during core hours (0700 to 1530 hours, Monday though Friday). The Service Provider completes the Routine Priority 3 SO within 30 calendar days of receiving the request. The Service Provider analyzes the scope of the IJO and determines the specific methods and resources needed, to include the number of hours, type of trade and material and equipment requirements to be used to accomplish the IJO. The Service Provider makes an investigation/inspection of the job site to determine the detailed requirements and to establish the operational sequences to complete the work. In the event of an unscheduled utility disruption, the Service Provider provides a verbal report to the KO or designee as soon as possible, but not later than one hour after utility disruption began. The Service Provider provides the suspected cause, affected facilities, and approximate service restoration time. In the event of a power failure, the Service Provider contacts and coordinates all work through the local power utility company system’s operator, check substations to confirm loss of power condition (total power failure, partial power failure, or single phase condition), and check backup generators for proper operation. The Service Provider performs all tests necessary to troubleshoot, make repairs, and restore power. 20
Slide 21: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Management and Administration PWS APL Standards 1.1.1.1 Timeliness MEASUREMENT PWS REF ACTIVITY STANDARD APL Begin work on 95% of Priority 3 SOs within five working days of receiving the request. Begin work on 100% of Priority 3 SOs within 10 working days of receiving the request. Complete 90% of Priority 3 SOs within 30 calendar days of receiving the request. Complete 100% of Priority 3 SOs within 60 calendar days of receiving the request. UNIVERSE (SOURCE) SURVEILLANCE METHOD C-5.2.2.2.3 Routine (Priority 3) Service Orders Begin work on Priority 3 SOs within five working days of receiving the request. Priority 3 SOs requests received in one month. Surveillance established by QASP and performed by CGA QAEs C-5.2.2.2.3 Routine (Priority 3) Service Orders The SP shall complete the Priority 3 SO within 30 calendar days of receiving the request. Priority 3 SOs requests received in one month. 21
Slide 22: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 MEASUREMENT PWS REF ACTIVITY STANDARD APL Coordinate, in writing, with the occupants of and activities/ organizations associated with the buildings/areas where the outages will occur and the KO or designee 95% of the time not less than 15 working days prior to the outage. Coordinate, in writing, with the occupants of and activities/ organizations associated with the buildings/areas where the outages will occur and the KO or designee 100% of the time not later than the 3rd working day prior to the scheduled outage. Annual scheduled outages UNIVERSE (SOURCE) SURVEILLANCE METHOD C-5.2.9.1 Planned Disruptions Coordinate, in writing, all scheduled outages or disruptions of utility services with the occupants of and activities/organizati ons associated with the buildings/areas where the outages will occur and the KO or designee work area(s) at least 15 working days prior to the outage. Surveillance established by QASP and performed by CGA QAEs 1.2 Facilities Maintenance and Repair The Service Provider performs scheduled maintenance tasks on a recurring or scheduled basis on all Facilities and Utilities systems listed in TE 5.6, Facilities Equipment Inventory by Building - with PM and TE 5.16, Utilities. The Service Provider maintains real property, installed property and equipment, and other designated equipment (TE 5.7, Real Property Inventory) located on DDJC (Tracy and Sharpe sites). Facilities Maintenance and Repair PWS Standards 22
Slide 23: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.2.1.1 Quality PWS REF ACTIVITY STANDARD APL On an annual basis, complete 90% of all PM IAW requirements set by manufacturer (new facilities systems) or for each existing facility system item displayed in TEs. MEASUREMENT (SOURCE) SURVEILLANCE METHOD UNIVERSE C-5.3.1 TEs 5.6, 5.11, 5.12, 5.13, 5.39 and 5.40 Scheduled Requirements (Facilities Systems Preventive Maintenance) Perform PM on facilities systems identified in TE 5.6, 5.11, 5.12, 5.13, 5.39 and 5.40 to ensure operational condition and availability to perform DDJC’s mission PM requirements for Facilities Systems listed in TEs and new facilities systems as established by manufacturer. Surveillance established by QASP and performed by CGA QAEs 1.2.1.2 Timeliness 23
Slide 24: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PWS REF ACTIVITY STANDARD APL On an annual basis, complete 90% of PM within frequency requirements set by manufacturer (new facilities systems) or existing facilities systems displayed in TEs. Complete PM on 100% of all facilities systems at least once every 2 years. UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD C-5.3.1 TE 5.6, 5.11, 5.12, 5.13, 5.39 and 5.40 Scheduled Requirements (Facilities Systems Preventive Maintenance) Complete PM within current PM schedule displayed in TEs 5.6, 5.11, 5.12, 5.13, 5.39 and 5.40 except for unavoidable delays approved by the KO or designee while awaiting parts. PM frequency requirements for Facilities Systems listed in TEs and new facilities systems as established by manufacturer. Surveillance established by QASP and performed by CGA QAEs 1.3 Utilities Systems Operation, Maintenance and Repair The Service Provider performs PM and operation of the water distribution systems, wastewater systems, high voltage systems, electrical secondary systems, natural gas systems, and associated structures and appurtenances at DDJC. The Service Provider performs services as required herein and as indicated in TE 5.16, PM Scheduled Utilities to provide complete operations and maintenance of utility systems. The Service Provider performs all work with qualified personnel IAW applicable laws, regulations. TEs provide expanded information for these systems. The Service Provider strives to provide efficient utility systems. Utilities Systems Operation, Maintenance and Repair PWS Standards 1.3.1.1 Quality 24
Slide 25: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PWS REF ACTIVITY STANDARD APL On an annual basis, complete 90% of all PM IAW requirements set by manufacturer (new utilities systems) or for each existing utility system item displayed in TE. 95% of Waste Water samples comply with the applicable sampling and testing procedures. UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD C-5.4 TE-5.16 Utilities Systems Operation, Maintenance and Repair Perform PM on facilities systems identified in TE 5.16 to ensure operational condition and availability to perform DDJC’s mission PM requirements for Utilities Systems listed in TE and new utilities systems as established by manufacturer Surveillance established by QASP and performed by CGA QAEs C-5.4.1.1 TE-5.18 Sampling and Laboratory Analysis Collect daily and weekly Waste Water samples to comply with applicable operating permit requirements and conditions and as directed by the KO or designee. Waste Water samples collected per quarter. Surveillance established by QASP and performed by CGA QAEs 1.4 Grounds Maintenance and Repair The Service Provider provides roads and grounds maintenance. Road/Paved area maintenance ensures that paved areas are clean and provide a safe driving surface. The Service Provider maintains grounds such as lawns, bedding areas, grass, trees, shrubs, plant beds and plants. DDJC uses four designations of grounds: improved, semi-improved, unimproved, and security right of way. The Service Provider edges, trims, prunes, top dresses, weeds, and mows IAW the PWS requirements. The Service Provider provides grounds maintenance services to all areas indicated in TE 5.27, Ground Maintenance Maps. Grounds Maintenance and Repair PWS Standards 1.4.1.1 Quality 25
Slide 26: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PWS REF ACTIVITY STANDARD APL 95% for improved grounds 90% for semiimproved grounds 85% for unimproved grounds UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD C-5.5 TE 5.29 Grounds Maintenance and Repair Perform grounds maintenance and repair to all areas indicated in the Grounds Maintenance Maps and TE 5.29. Monthly maintenance and repair requirements for areas indicated in the Grounds Maintenance Maps and TE. Surveillance established by QASP and performed by CGA QAEs 1.5 Equipment Maintenance and Repair Equipment maintenance includes both preventive and corrective maintenance. DDJC has approximately 1300 pieces of equipment owned by various organizations (e.g., installation-owned, General Services Administration (GSA)-leased, and other vehicles and equipment). The term “equipment” as used herein includes but is not limited to MHE, motor vehicles (wheeled or tracked), heavy mobile equipment, construction equipment, other motor vehicles such fire equipment vehicles, and other equipment. GSA vehicles are not included in the term “equipment,” and will be discussed separately below. The Service Provider performs equipment maintenance and repair IAW: • 41 CFR, Public Contracts and Property Management, Subtitle C, Federal Property Management Regulations System, Chapter 101, Federal Property Management Regulations (FPMR) 41 CFR, Public Contracts and Property Management, Subtitle C, Federal Property Management Regulations System, Chapter 102-34, Motor Vehicle Management DLA One Book, Chapter: DLA Enterprise Support, Title: Environmental Compliance DLA One Book, Chapter: DLA Enterprise Support, Title: Safety DLA One Book, Chapter: DLA Enterprise Support, Title: Support Disposal Process DLA One Book, Chapter: DLA Enterprise Support, Title: Support Equipment Maintenance Process DoD 4500.36-R, Management, Acquisition, and Use of Motor Vehicles, Chapter 12, Maintenance Management; EMACS schedules Manufacturer’s specifications • • • • • • • • 1.5.1 Equipment Maintenance and Repair PWS Standards 1.5.1.1 Quality 26
Slide 27: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PWS REF ACTIVITY Maintain Availability Rate for Category Auto including: Bus (38+ Passenger, Bus (37- Passenger), Fire Trucks, Heavy Trucks, Light Trucks, Trailers, Emergency Vehicles, Passenger Vehicles Maintain Availability Rate for Category Major Equipment including: Installation Cranes Maintain Availability Rate for Category MHE including: Elec Forklifts, IC Forklifts, Stock Forklifts, non Metered MHE, Whse Tractors Maintain Availability Rate for Category Miscellaneous including: Misc Whse Equip, Sweepers/Scrub STANDARD APL UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD C-5.6 Maintain auto availability. 95% of the time Monthly EMACS Utilization and availability report for the Auto Category Surveillance established by QASP and performed by CGA QAEs C-5.6 Maintain Major Equipment availability. 95% of the time Monthly EMACS Utilization and availability report for Major Equipment Surveillance established by QASP and performed by CGA QAEs C-5.6 Maintain MHE availability. 95% of the time Monthly EMACS Utilization and availability report for MHE Surveillance established by QASP and performed by CGA QAEs C-5.6 Maintain Miscellaneous Equipment availability. 95% of the time Monthly EMACS Utilization and availability report for Miscellaneous Equipment Surveillance established by QASP and performed by CGA QAEs 27
Slide 28: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.5.1.2 Timeliness PWS REF ACTIVITY STANDARD APL 90% of equipment PM is completed in ≤ 2 days of receipt of equipment 100% of equipment PM is completed in ≤ 5 days unless prior approval received from KO or designee 95% of equipment repairs are completed in ≤ 7 days of receipt of equipment. 100% of equipment repairs are completed in ≤ 10 days unless prior approval received from KO or designee. UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD C-5.6.2.1 TE-5.36 Equipment PM Perform PM on vehicles and equipment and return to customer. Monthly EMACS PM report with date received and date completed Surveillance established by QASP and performed by CGA QAEs C-5.6.2.1 TE-5.36 Equipment Repairs Complete Unscheduled Repairs and return to the customer unless awaiting materials and approved by the KO or designee. Monthly EMACS maintenance report with date received and date completed Surveillance established by QASP and performed by CGA QAEs 28
Slide 29: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PWS REF ACTIVITY STANDARD APL 90% of trouble calls are responded to in ≤ 30 minutes of receipt of notification from the Government for Tracy. 100% of trouble calls are responded to in ≤ 60 minutes of receipt of notification from the Government for Tracy. 90% of trouble calls are responded to in ≤ 2 hours of receipt of notification from the Government for Sharpe 100% of trouble calls are responded to in ≤ 4 hours of receipt of notification from the Government for Sharpe. UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD C-5.6.3.1 TE-5.29 Trouble Call response Respond to trouble call (breakdown of MHE/Non-GSA/GSA vehicle. Monthly EMACS maintenance report with date received and date completed Surveillance established by QASP and performed by CGA QAEs 29
Slide 30: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PWS REF ACTIVITY EMACS Records updated STANDARD APL NLT the next working day of work order completion 95% of the time. UNIVERSE MEASUREMENT (SOURCE) SURVEILLANCE METHOD Surveillance established by QASP and performed by CGA QAEs C-5.6.1 EMACS updated unless system is down. Monthly EMACS maintenance report with date received and date completed 1.6 Mail Services Mail services shall be provided IAW DoD 4525.8-M, DoD Official Mail Manual; DoDI 4525.8, DoD Official Mail Management; the USPS Domestic Mail Manual (DMM); and the International Mail Manual (IMM). The Service Provider’s Mail Center works in conjunction with the USPS and adheres to USPS DMM and IMM guidelines. The Service Provider provides as a minimum the operating hours of 07001530, Monday – Friday. The Service Provider pick-ups and delivers official mail, accountable mail, bulk mail, inter-office correspondence, and parcels to sites throughout the installation as indicated in TE 5.37, Mail Pickup and Delivery Points, and includes delivery to only the Tracy site within 24 hours of receipt by the Mail Center. No deliveries shall be made to the Sharpe site. Mail Services PWS Standards 1.6.1.1 Quality PWS REF ACTIVITY STANDARD Receive and accurately distribute mail within 24 hours unless the KO or designee has approved delays for unexpected events such as severe weather APL MEASUREMENT (SOURCE) SURVEILLANCE METHOD Surveillance established by QASP and performed by CGA QAEs UNIVERSE C-5.7.1.1 TE-5.37 Mail Distribution Services 98% Number of pieces of mail received each month in the monthly production report 1.7 Non-APL Contract Mandatory Requirements The value and importance of assuring Service Provider performance of the APLs outlined in the contract cannot be over stated. While the vast majority of the APL criteria contained in the contract are based on timeliness criteria, the most complete evaluation of the Service Provider’s performance relies on other criteria as well. Coupled with timeliness criteria are: • • • Process Quality Adherence to Policies and Procedures (administrative and process) Security 30
Slide 31: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • • • • • Safety Training Maintenance and Accessibility of Records and Documents Maintenance of Government Furnished Equipment Financial Information Customer Satisfaction These, as well as other unlisted areas (such as “shall” requirements), influence the Service Provider’s achievement of contract requirements and customer satisfaction. As indicated in the Federal Acquisition Regulation (FAR), “shall” is an imperative – a “mandatory” requirement. Therefore, these requirements effectively have an APL of 100% attached to them, even though there are not any “APL Tables” included in the contract (for ease of discussion, these requirements are called “non-APL” requirements). As specified in the solicitation, the Service Provider remains responsible for providing a mechanism to control all the requirements contained in the contract with implementation of a QC/CSP prepared in accordance with Section L-15 of the RFP. The Service Provider’s QC/CSP must comply with FAR 52-246-4(b), which states: The Service Provider shall provide and maintain an inspection system acceptable to the Government covering the services under this contract. Complete records of all inspection work performed by the Service Provider shall be maintained and made available to the Government during contract performance and for as long afterwards as the contract requires. This requirement directs the Service Provider to provide an “inspection system,” including the records of the inspection work, for the services covered by the contract. Therefore, the Service Provider’s QC/CSP should address both APL and “non-APL” requirements. The Government must monitor these areas in order to assure the Government that the Service Provider remains in compliance with contract requirements. To monitor these activities effectively, the Government must exhibit a physical presence in the process areas in order to perform the following: • • • • • Conduct surveillance/audits of product quality Monitor compliance to policies and procedures by observation of employees in the performance of tasks Evaluate compliance to the Service Provider’s QC/CSP Audit documentation of process inspections, maintenance records, security logs and training records Evaluate records and documentation currently in use at the point of task completion and those archived for historical records retention Before proceeding further in this section, understanding a few terms commonly used in the Quality Field remains important. Therefore, listed below are five terms, with definitions, taken from J. M. Juran’s 31
Slide 32: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Quality Control Handbook. These five functions remain the foundation of the responsibilities of the Government in assuring that the Service Provider complies with contract APLs and requirements. Process Quality Audit: An analysis of elements of a process and appraisal of completeness, correctness of conditions, and probable effectiveness. Product Quality Audit: A quantitative assessment of conformance to required product characteristics. Quality Audit: A systematic and independent examination and evaluation to determine whether quality activities and results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives. Quality System Audit: A documented activity performed to verify, by examination and evaluation of objective evidence, that applicable elements of the quality system are suitable and have been developed, documented, and effectively implemented in accordance with specified requirements. Quality Surveillance: The continuing monitoring and verification of the status of procedures, methods, conditions, products, processes, and services, and analysis of records in relation to stated references to ensure that requirements for quality are being met. The following pages discuss various areas and functions audited for compliance to contract requirements. E, Helps and Aids, of this QASP provides sample forms, checklists, reports and other information useful to Government Surveillance Specialists. The checklists are starting points for the surveillance/audit to which other criteria may be added. The checklists are adaptable to individual situations and circumstances. 1.7.1 Using Random Sampling in Surveillance of Non-APL Requirements When the Service Provider monitors Non-APL requirements through implementation of a QC/CSP that documents and reports performance of these Non-APL requirements, the Government is able to use one of several “less than 100%” surveillance methodologies to validate the Government’s performance. As indicated above, the Surveillance Method Selection Tool, Random Sampling, Periodic Surveillance/Monitoring and User Surveys are acceptable methods for performing surveillance. However, the Government should only use Random Sampling as part of scheduled surveillance of Non-APL requirements. When conducting random sampling, the process to determine the acceptance or rejection limits for Non-APL requirements monitored in the Service Provider’s QC/CSP is a little more complex than those specified for APL requirements since “standards,” “sources” and “measurement universes” are not established. The following steps outline the process: a) Determine the lot size. Use the results documented by the Service Provider in execution of its QC/ CSP for a specific Non-APL activity as the lot size. 32
Slide 33: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 b) Determine the sample size. Based on the lot size, determine the respective sample size for normal inspection by referring to Figure 0-3, Determine the accept/reject values. As a rule, non-conformances are divided into three classifications categorized as Minor, Major, and Critical. The definitions are contained in paragraph Nonconforming Outcomes, Page 11. c) Note: The DDJC (or the DDC) must specify the percentage of defective product/services the DDJC willingly accepts and/or the percentage of acceptable product/services the DDJC willingly rejects based on statistical sampling. For example, if the DDJC accepts a 4% AQL, this means that the DDJC willingly accepts defective product/services 4% of the time based on statistical sampling. At the same time, the DDJC must realize that good product is turned away or rejected 4% of the time. LOT SIZE 2-8 9-15 16-25 26-50 51-90 91-150 151-280 281-500 501-1,200 1,201-3,200 3,201-10,000 10,001-35,000 35,001-150,000 150,001-500,000 500,001-Over NORMAL INSPECTION SAMPLE SIZE 2 3 5 8 13 20 32 50 80 125 200 315 500 800 1250 CRITICAL AQL MAJOR AQL 0.065 0.40 ACCEPT REJECT ACCEPT REJECT 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 1 2 0 1 2 3 0 1 3 4 1 2 5 6 1 2 7 8 2 3 10 11 Figure 0-3, Random Sampling Plan MINOR AQL 6.5 ACCEPT REJECT 0 1 0 1 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22 21 22 21 22 21 22 21 22 The chart above (Figure 0-3, ) indicates that with a sample size of 125 (lot size between 1,201 – 3,200) and one major nonconformance and 14 minor non-conformances, the lot is acceptable. However, if using a single sampling plan of 0.40, regardless of the category of nonconformance, the lot is rejected if there are only two non-conformances. Another helpful tool is to assign a Probability of Recurring Code (PRC) (see Figure 0-4, Probability of Recurring Codes) to reports and correspondence concerning quality assurance findings. These codes alert the recipient to the severity of the problem reported. The matrix below (Figure 0-4, Probability of Recurring Codes) illustrates the PRC and the relationship to the risk factors identified above. RISK FACTOR Critical PROBABILITY OF RECURRING PROBABLY MAY CR-2 CR-3 33 LIKELY CR-1 UNLIKELY CR-4
Slide 34: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Major Minor MA-1 MA-2 MA-3 MI-1 MI-2 MI-3 Figure 0-4, Probability of Recurring Codes MA-4 MI-4 The PRC functions as a visual indicator or flag to ensure that critical, major, and minor findings are addressed in a timely manner, indicative of the severity of the situation. The PRC initiates predetermined procedures for root cause, corrective action, and prevention plan development based upon the risk factor coupled with the probability of recurrence. The following outlines the key Service Provider/Government action requirements associated with the PRCs CR-1 and MI-4. Actions for all other PRCs lie between these two PRCs and are affected by management’s risk-cost-benefit analysis. Note: Unless other wise indicated, the corrective actions are Service Provider responsibilities. Followup to ensure actions correct the problem are both Service Provider and Government responsibilities. • CR-1: Critical with a likely probability of recurring  Take immediate actions to eliminate any existing potential danger  Take immediate action to prevent defective product from reaching the customer  Immediately advise management of the Government and Service Provider of the situation.  Begin same day root cause and corrective action investigation  Implement Fast Track corrective action in 24 hours maximum  Determine preventative action  Implement preventative action plan  Verify effectiveness of corrective and preventative actions (by Service Provider & Government)  Adjust corrective and preventative action plans as needed to maintain effectiveness MI-4: Minor nonconformance with an unlikely probability of recurrence  Document findings  Report findings in accordance with normal reporting procedures Other Surveillance Methods for Use in Non-APL Requirements • 1.7.2 If the Service Provider’s QC/CSP does not address Non-APL requirements as part of its Quality Control/Customer Satisfaction Plan methodology, the Government may use One Hundred Percent (100%) Inspection, Periodic Surveillance/Monitoring and User Surveys for its scheduled surveillance activities (see the Surveillance Method Selection Tool for help in selecting the appropriate surveillance method). For unscheduled surveillance of Non-APL requirements, Direct Observation/Random Visual Surveillance/Monitoring, One Hundred Percent (100%) Inspection, Periodic Surveillance/Monitoring and User Surveys are all surveillance methods available for Government use. 1.7.3 Monitoring the QC/CSP – Process Quality 1.7.3.1 Review of the QC/CSP during Phase-In The Government is responsible for ensuring that the Service Provider’s QC/CSP provides a documented quality system that is effective in achieving contract requirements. To validate QC/CSP 34
Slide 35: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 effectiveness, the Government reviews the QC/CSP and makes an appraisal of completeness, correctness of conditions, implementation, and probable effectiveness. The Government may evaluate the QC/CSP at anytime; however, assessment of the QC/CSP is particularly important during a new Service Provider’s phase-in period and as full performance begins. Some, but not all of the factors evaluated by the Government, are as follows: • • • • • • The quality system addresses all distribution process and monitors the processes for quality. The operator makes in-process quality checks. The Service Provider documents in-process quality checks. Samples taken represent a statistically significant number to verify the quality system. The Service Provider conducts continuous sampling and monitoring of the quality system. The quality system provides for a quality team independent of production, which reports to the Service Provider’s management. The goal of the review is to help the Government ensure that the Service Provider successfully performs contract requirements, including quality-control tasks. The Government has the option of inviting the Service Provider to participate in the QC/CSP review. Whether or not the Service Provider participates in the review, following the review, the Government submits any recommendations for improvement of the QC/CSP to the Service Provider, COR and KO. 1.7.3.2 Review of the QC/CSP for Quality Issues When it becomes evident that the Service Provider failed to meet contract APL and Non-APL requirements in a particular area, the Government may conduct a targeted assessment of the QC/CSP. Targeting two aspects of the QC/CSP usually successfully determines a viable solution to the problem. The Government may conduct these targeted reviews in conjunction with or independent of the Service Provider. The first area to validate is whether the employees perform tasks in compliance with the established processes as established in the QC/CSP. If the employees do not follow the designated QC/CSP processes, this could indicate the potential root cause. However, if the employees comply with QC/CSP procedures, then the second targeted area determines the effectiveness of the process documented in the QC/CSP. If the process documented in the QC/CSP does not appear effective, the Service Provider may need to consider revising the process. 1.7.4 Product Quality The Service Provider remains responsible for product/service quality, achieved through strict adherence to the QC/CSP. Verification of product/service quality is a responsibility of the Government. The Government verifies product/service quality by a combination of the following activities: • Examination of Service Provider in-process and archived quality records 35
Slide 36: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • Review of IFS or EMACS records Examination of the product/service during a Product/Service Quality Audit  planned or unplanned audits  conducted in any function from receipt to shipment and training to transportation Contacting the customer for feedback on product/service quality, (i.e., by phone, mail or e-mail, meetings, conferences, etc.) • 1.7.4.1 Triggers for Conducting Product/Service Quality Audits Various factors, listed below, come into play in determining which product and when to conduct a product audit: • • • IFS records, which indicate potential problem areas Customer complaints TBD It is not practical to examine every aspect of any particular service or product. Nor is it practical to examine every item in a particular lot. Therefore, the Government must make judgments based on circumstances and situations. 1.7.4.2 Trigger Examples TBD (Develop Installation Services Trigger Examples) 1.1.1.1 Situation 1.1.1.2 Situation 1.1.1.3 Situation 1.7.5 Policies and Procedures The Service Provider is responsible for ensuring their employees understand and follow all policies and procedures as outlined in the contract, including the QC/CSP. Policies and procedures are set in place to ensure consistency and predictability within a process and the organization. Confidence in the system’s ability to produce quality consistently weakens by permitting personnel to deviate from established policies and procedures. Therefore, it becomes important for the Service Provider to enforce policies and procedures, and equally important for the Government to validate enforcement through auditing and monitoring. To monitor policies and procedures effectively the Government must be knowledgeable of the policies and procedures outlined in the contract, including the QC/CSP. “Knowledgeable” indicates not only formal training of the contract, including the QC/CSP, but also ongoing individual review and study of these documents. For example, by possessing a working knowledge of the QC/CSP procedures, the Government knows at what stage of the process and at which workstations a particular function occurs. With knowledge 36
Slide 37: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 of contract policies and procedures, the Government understands that freight drivers do not enter a security cage where classified items reside. 1.7.5.1 Monitoring Policies and Procedures Monitoring Service Provider compliance with contract policies and procedures is an ongoing daily task. It is part of knowing and understanding the contract, including the QC/CSP. As the Government performs other surveillance duties, a positive “situational awareness” helps ensure Service Provider infractions of policies and procedures become self-evident. With violation identification, the Government takes steps to document and inform responsible personnel for corrective action. Monitoring policies and procedures can also be a planned activity. Planning audits of a particular policy or process becomes prudent with the observation and reporting of multiple past infractions. The audit becomes a means for verification of effective corrective actions. 1.7.6 Security In today’s world of “post 9/11,” security of facilities and DDJC materials, equipment and personnel are of paramount importance. The contract requires that the Service Provider remain diligent in providing a secure environment for depot personnel. The Government must confirm that the employees, the facility and the materials in the DDJC remain secure from subversive activities and/or acts of security violations from within. The contract covers the subject of security in detail and establishes the policies and procedures the Service Provider must use for installation access, visitor control, key and lock control, and security of classified materials. The execution of the Government’s security responsibilities highlight the fact that monitoring of security compliance is an important function. 1.7.6.1 Security Audit Criteria A comprehensive understanding of the requirements of the contract is important in planning an audit of security compliance. The Government must understand the intent, as well as the language, of the contract. Knowing the intent of security may promote a questioning attitude on the part of the Surveillance Specialist when something “just doesn’t seem right.” Planned audits may focus on a particular aspect of security compliance, such as exterior gate control. Audits may also be broader-based and look at all aspects of security in a particular area or facility. Some security criteria that require consideration for incorporation into a security audit are included in the questions below: • • • • • • Do all employees and visitors have visible security (ID) tags? Does the Service Provider escort visitors at all times? Do Service Provider personnel share PIN or Card Access Badges? Are controlled inventory items properly identified, segregated, stored in secure locations? Are required secure locations locked? Is access to secure locations restricted to authorized personnel only? 37
Slide 38: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • • • • Are secure locations logs completely filled out, signed and dated? Are secure locations logbooks reviewed and signed by supervisory personnel on a timely basis? Are lights burnt out in areas that should be well lighted such as parking lots, ingress and egress points? Is key control effective? Is there any evidence of pilfering/theft? Safety 1.7.7 The Service Provider’s responsibility to maintain a safe work environment is mandated by the contract, federal, state, and local laws and regulations. The Government is in a unique position to observe unsafe conditions and behavior in a wide variety of locations throughout the workday. Safety of employees is an overriding concern in all operations at the DDJC. Therefore, the Government brings safety issues to the attention of responsible personnel immediately as they arise. The Service Provider’s responsibility is to act upon the information in the form of corrective and preventative actions. 1.7.7.1 Safety Audit Criteria Since the contract provides specific guidance in a number of safety areas requiring Service Provider compliance, the Government must ensure that these requirements are properly fulfilled. Safety audits, performed by Surveillance Personnel as part of routine surveillance activities, provide an excellent method for ensuring Service Provider compliance with safety requirements. Not every member of the Government is expected to be a safety expert knowledgeable in all aspects of industrial and product/services safety. Each Government, however, does have a safety/security program manager with direct access to the DDC Safety and Security Program Managers. Other Surveillance Specialists are easily able to perform surveillance of safety and security related issues using checklists such as those provided in this QASP, as well as relying on direction from the Government Safety/Security Program Manager. They are also able to integrate depotwide safety checks during the performance of daily surveillance tasks. The list below contains many of the safety issues discussed in the contract which represent candidates for surveillance audits: • Environmental  Hazardous substance spills  Training records  Proper storage of flammable and combustible materials  MAPL on file for hazardous materials Fire Safety      • Fire extinguishers present at all locations Discharged fire extinguishers replaced No accumulation of combustible materials No smoking in unauthorized places Fire Evacuation Plan available • Work Practices 38
Slide 39: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11      1.7.8 Required protective clothing and equipment used Proper behavior on the floor Pallets stacked appropriately Notification of safety hazards in work area Holding safety meetings Training As part of the contract requirements for training, the Service Provider provides training for Service Provider personnel and maintains training records, which include, at a minimum, the name of the employee, the name of the course, the source of the training, a description of the training provided and the date the employee successfully completed the training. The contract requires that the Service Provider provides the following training to employees as applicable: • Asbestos • Compressed Gas Cylinders • Confined Spaces • Environmental Protection Agency (EPA) Stratospheric Ozone Certification (MVAC) • Equipment Operation and Maintenance • First Responder Awareness Level • Hazard Communication • HAZMAT Transportation • HAZWOPER • Information Assurance • Lead Exposure • Oil Pollution/Oil Recycling • Ozone Depleting Substances • Pesticide Applicator Certification • Respirators • RCRA for Generators of HW • Security • Supply 2000 (S2K) • X-Ray In addition to Service Provider-furnished training, Service Provider personnel also receive training provided by the Government as indicated in the table below: Government-Furnished Training and/or Training Material System DPAS EMACS IFS Method Of Training Government Instructor Government Instructor Government Instructor Site On-site Off site (TBD) On site Length 1 day Four days Authorized Sp Employees To Attend As required1 Two As required1 Completion Date End of phase-in2 End of Phase-in2 End of phase-in Frequency One-time One-time One-time 39
Slide 40: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Government-Furnished Training and/or Training Material System IPMIS Storage and Handling of HAZMAT S2K Method Of Training Government Instructor Government Instructor Site On-site On-site Length Authorized Sp Employees To Attend As required1 As required1 Completion Date End of phase-in Performance start date + 90 days Frequency One-time One-time 2 days Government On-site As required End of Phase-in One-time Instructor 1 “As required” denotes that the SP is authorized to send as many employees to the training as it determines is necessary to meet the PWS Acceptable Performance Levels (APLs) or to comply with the contract/LoO/Agreement requirements. 2 Or the first available date for government-provided training Note: The list of Government-Furnished Training is included in the RFP. Due to health and safety risks and the security and criticality of specific tasks, the Government determined that individuals performing in these job functions require training. The Government audits training records to ensure complete and up-to-date, required information on the records. The audit also verifies that an employee performing a particular job function received the training required by the contract to perform the job. 1.7.9 Records and Documents To comply with the contract, the Service Provider must ensure the accuracy and completeness of all documents and records. The Government uses records and documents as key indicators of Service Provider compliance with policies and procedures, since the documents and records provide historical and traceable data. To achieve the intended purpose, records and documents must be complete, displaying no empty blocks or incomplete information. Blank entries or incomplete information often raise more questions than they answer. To assure the Government of complete and accurate records, the Surveillance Specialist must audit Service Provider records and documents. All Service Provider records and documents are candidates for inclusion in an audit, including electronic and hardcopy documents. The examples below list documents and records included in audits: • • • • • • Training Records Certification Records Inventory Records QC/CSP Process check sheets Quality Control/Customer Satisfaction Plan documents 40
Slide 41: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • • Equipment Management and Control System (EMACS) Operator safety check sheets Decision-making e-mails 1.7.9.1 Records and Documents Audit Criteria The record and document audit includes documents currently in use at workstations, as well as archived documents. The audit verifies the following: • Are records complete?  Are all blocks filled in? If a block does not apply a “–“(dash) or “N/A” is inserted in the block.  Are all records dated? Is the date a complete date (i.e., are the day, month and year included?)  Are all signature blocks signed? Are corrections properly made?  Is liquid paper, correction tape, or correction fluid use to correct mistakes? If yes, the correction has not been properly made.  Are errors indicated by drawing a single line through the words?  Is the line-through initialed and dated by the individual making the correction?  Is the correct information inserted above or below the lined-through error? Are there missing documents?  If a particular document is completed daily, weekly or monthly, are all documents present without a break in the chronology? Are records protected from loss, damage or unauthorized access or use? Are records accessible only to authorized individuals? Maintenance and Care of Government Furnished Equipment and Facilities (GFE and GFF) • • • • 1.7.10 The contract outlines the Service Provider’s responsibility for care and maintenance of GFE, such as Powered Industrial Trucks and other MHE. The contract requires the Service Provider to use EMACS to track repair and maintenance on GFE (see Section C-3 of the PWS). EMACS is subject to audit as outlined in paragraph C.1.11.9, Records and Documents above. Other hardcopy documents related to the maintenance and care of GFE require review to provide the most complete picture of the Service Provider’s compliance with maintenance and care of GFE and GFF. These documents may include daily and weekly safety/maintenance check sheets or maintenance work orders. The Government conducts record or observation audits or a combination of records and observations to assure compliance to equipment maintenance and care. The Government does not routinely monitor major repair activities to verify proper repair of GFE of GFF. However, the Government checks records and verifies work completion. If other triggers, such as 41
Slide 42: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 complaints of inoperative equipment, observations of large quantities of “dead” equipment, observations of damage to equipment or facilities exist, the Government initiates a more in-depth monitoring of GFE repair activities. The Government observes startup of equipment at the beginning of shift operations to verify the completion of pre-startup checks on individual equipment. The Government may detect malfunctioning safety equipment, such as flashing lights, backup horns, or the lack of use of safety belts and lanyards by simple observation. The audit plan for maintenance and care of GFE may easily incorporate these observations and others, such as instances of equipment cannibalization. Note: The Government remains responsible for maintenance of GFF; however, the Service Provider is responsible for notifying the Government of facility maintenance requirements. The Service Provider is also required to maintain existing storage racks, moveable racks, carousels, storage aids, and bins located within the GFF. Additionally, the Service Provider is not authorized to construct or install, at the Service Provider’s expense, any fixed improvement or structural alterations in government buildings or other real property without advance written approval from the KO. Observant Surveillance Specialists, while performing daily activities within the warehouse and work areas, can detect damage caused by careless use of MHE, or identify needed repairs or unapproved facility remodeling activities. The Government monitors and reports such activities to Government management. 1.7.11 Customer Satisfaction The Service Provider is responsible for ensuring customer satisfaction with Service Providerprovided products and services. “Not receiving complaints” is not necessarily a true indicator of customer satisfaction. Many customers may not respond to unsatisfactory performance by completing a User Complaint Report or even by reporting dissatisfaction with a product or service. As a more objective measure of ensuring customer satisfaction, the Service Provider must proactively engage the DDJC’s customers for feedback on performance. The Service Provider may take a number of avenues when judging the customer’s overall satisfaction; all of which require documentation. The following examples represent proactive steps the Service Provider can take: • • Call the customer after a shipment delivery to inquire on its condition upon arrival (document the response) Send the customer survey cards with coded questions to gage the customer’s satisfaction level with various elements of service and product conditions (maintain an archive of returned cards and perform analysis of the responses) Track User Complaint Records to detect and correct recurring, out-of-control problems Request and implement, as appropriate, suggestions for improvement from the customer. • • The Government is responsible for auditing the Service Provider’s customer satisfaction program and reporting the results of the audits to the DDC. Regular surveillance of the customer satisfaction program reveals the program’s effectiveness as a tool to improve customer communication and satisfaction. The following questions represent some key indicators when auditing the Service Provider’s customer satisfaction program: 42
Slide 43: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • • • • Is there documented evidence of Service Provider communication with the customer dealing with improved customer service and satisfaction? Is communication on a regular basis? Is there evidence of analysis of customer comments and suggestions? Are corrective actions taken to improve customer satisfaction? Does the Service Provider inform employees of customer issues? Documentation Requirements Section D. 1.1.1 Contract File Records One of the foundations of QASP implementation is documentation of performance assessment activities. These Quality Assurance records are maintained as part of the “contract administration office contract file.” (See FAR 4.803, Contents of Contract Files.) The FAR (Subpart 4.802) requires contract files to be maintained in a method to ensure: • • • • • Effective documentation of contract actions Ready accessibility to principal users Minimal establishment of duplicate and working files The safeguarding of classified documents; and Conformance with agency regulations for file location and maintenance Key among the Contract Administration functions is the Administrative Contracting Officer’s (ACO) responsibility to ensure compliance with contractual quality assurance requirements. The contract file, including documents held in all types of media (e.g., paper, electronic, microfilm, etc.), is the ACO’s record to document the selected Service Provider’s compliance. The contract file, as required by FAR Part 4, must be maintained for a minimum of six years and three months after final payment. Since this file includes documents developed and submitted to the contracting office as part of the Government’s performance assessment activities, the Government must establish a contract filing system that meets all FAR requirements. Additionally, easy-to-use and complete documents, good surveillance-team management, and a disciplined and diligent team in filling out the required documentation are “musts” for successful performance documentation. Anecdotal evidence and personal opinion do not provide a legal basis for accepting or rejecting Service Provider performance. Timeliness, accuracy, and comprehensiveness of information are required to document both satisfactory and unsatisfactory performance. 43
Slide 44: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.1.2 Rules of Documentation When filling out Surveillance Reports, or any other documents used during monitoring, analyzing, or reporting, adhering to the “Rules of Documentation,” listed below, remains important: • • Use only black or blue ink. Fill in all informational blocks and blanks on the form. If a particular block does not apply, insert “N/A.” A blank informational block when reviewed months later may raise questions as to whether the missing information applies or not. Never erase or use correction fluid, correction tape, or liquid paper on a document. If an error occurs,  draw a single line through the error;  initial and date the line–through; and  insert the correct information above or below the line-through. Always sign and date the documents. Access to Service Provider Records As required by Section C-6 of the contract: The Government will have unlimited rights to use, duplicate, or disclose Service Provider’s operating logs and forms, in whole or in part, in any manner, for purposes associated with execution of this award......Records maintained by the Service Provider remain the property of the US Government and will be retained IAW disposition instructions. The following list provides samples of the Service Provider’s documentation, which must be available for Government review. • QC/CSP implementation records  Quality Control/Customer Satisfaction Plan Check results  QC/CSP schedules  Customer feedback records Employee Training and Certification Records  Government-furnished training, as specified in the contract  Local, State and Federal required training Records demonstrating the Service Provider’s maintenance of Government Furnished Property  Calibration and certification records  Equipment maintenance and operational safety certifications Property control procedures and records Service Provider payroll and other records to substantiate reimbursement requests for labor hours, material, other direct costs, travel and per diem • • 1.1.3 • • • • 44
Slide 45: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • 1.2 Visitor Control Logs Environmental Compliance Records Surveillance Reports The Service Provider Surveillance Report (SR) provides a summary of surveillance information and is the basic form used to document surveillance and monitoring of Service Provider performance for both APL and Non-APL contract requirements. Use Service Provider Surveillance Reports for both scheduled and unscheduled surveillance. While it may appear that the format of the report tends to focus on unacceptable performance, the surveillance report actually assumes that most surveillance efforts identify satisfactory Service Provider performance; however, commendable and noteworthy performance also requires documentation. Filling out the Service Provider Surveillance Report is self-explanatory; therefore, complete blocks as indicated in the report form (see also 11.1, Surveillance Examples, for examples of how the Surveillance Report is used). Surveillance activities may rely on a single method or a combination of two or more surveillance techniques. The Government’s dedication to achieve the most complete and objective recognition and documentation of the Service Provider’s compliance or non-compliance with the APL or Non-APL requirements of the contract requirement governs the surveillance method or methods used. 1.3 User Complaint Record (UCR) The User Complaint Record is a tool for recording and documenting customer complaint investigations. The UCR is self-explanatory regarding the information needed to complete the form. The Government or the originating party who initially identifies the problem detailed on the UCR form completes the “representative” block in the Complaint Source Information Section of the form. The UCR documents, in sufficient detail, the pertinent information necessary to allow a complete investigation. When the Government initiates or receives a UCR, the Government records the UCR in a surveillance log. When appropriate, the Government maintains a copy of the UCR with the surveillance log. The following outlines the information detailed on the UCR: • • • • • • • Detailed information of the UCR originator (e.g., name, title, organization, phone, etc.) Date and time the UCR generated Name and date of the Government representative receiving the UCR Detailed description of the specific complaint Standard or requirement violated (e.g., contract reference or APL) Investigation surveillance log reference number Service Provider representative name, title, date and time informed of the original complaint and investigation results 45
Slide 46: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 • • Response and actions taken by the Service Provider, with attached supporting documentation Signature and date of the UCR preparer Determining the validity of performance violations may be a subjective decision during the verification of User Complaints. The Government (KO (or equivalent) and COR) must carefully evaluate the complaint to guard against unfair user opinions. In addition to providing the details of potential customer dissatisfaction, the evaluations may identify areas where user expectations are different from those defined in the contract or where inadequate communication occurs between the Service Provider and the user. Section E. Helps and Aids 1.1 Reports TBD (Develop example reports for Installation Services) 1.2 Phase-In Checklist The Phase-In Checklist below provides a tool used to ensure the understanding of and compliance to all key elements and requirements of the RFP. The three phases of the checklist follow: • • • Phase 1 – Post-Award Conference (begins at contract award or cancellation of solicitation) Phase 2 – Phase-In (begins on contract start date) Phase 3 – Post-Phase-In (begins at start of first performance period) TBD (Develop Checklists for Installation Services) COMPLETED/CERT/ SCHEDULED YES NO PHASE-IN CHECKLIST Post-Award Conference: (Phase I: To begin at contract award/cancellation of solicitation or as determined by the KO) 1. Review Service Provider Phase-In Plan 2. Review contract requirements. 3. Discuss Phase-In checklist and Phase-In POAM. 4. Receive notice about start of Phase-In period. 5. Discuss Phase-In Period requirements. 6. Receive tentative schedule for Government-furnished training. 7. Discuss Service Provider Additional Requirements 8. Address Service Provider concerns and issues. 9. Address the office space and equipment requirements that the Government needs to furnish to effect and complete the Phase-In plan. 46
Slide 47: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PHASE-IN CHECKLIST 10. Address any Government required actions. Phase-In: (Phase II: To begin at Contract start date) Start of Phase-In Period: 11. Provide in writing to the KO the name and telephone number (home, cell, office) of the contract manager and an alternate. 12. Government will provide the GFM on hand. 13. Submit an essential personnel list to the KO. 14. All publications and forms listed made available. During Phase-In Period: (Phase-In Period): 15. Submit a revised Quality Control/Customer Satisfaction Plan 16. Provide written report weekly to the KO on the status of Phase-In. Conduct/Receive Training: 17. IFS or EMACS: (24 hours) 18. Security Training: Materials provided; Service Provider provided (3 Hours) 19. HAZMAT Preparer Certification: All HAZMAT personnel; Service Provider provided 20. EMACS: (24 hours) Security Clearances, Badges/Decals, and Base Communications: 21. Obtain all licenses, certifications & security clearances as required in the RFP 22. Ensure proper application of National Agency Checks 23. Request ADP clearances for employees requiring access/passwords to the Government-furnished data systems. 24. Obtain a signed document for the acceptance and custody of all GFP keys 25. Ensure that every employee possesses a proper identification badge prior to starting work. Inventory: 26. Complete GFP Joint Inventory by Phase-In End Date minus 60 Days. 27. Submit the list of Service Provider-accepted equipment/assets by Phase-In End Date minus 60 Days. 28. Provide Service Provider with a final equipment/asset list from DPAS records by Phase-In End Date minus 5 Days. 29. End of Phase-In Period: 30. Maintain accountability of all GFP using the Defense Property Accountability System (DPAS). 31. Submit completed training documentation and/or certifications to the KO. 32. Meet or exceed the APL requirements as specified in contract. 33. Provide and maintain properly trained, qualified, and certified personnel. 34. Accept the GFP and GFRP for use and maintenance. 35. Schedule no inventories during the last 30 days of the Phase-In period. COMPLETED/CERT/ SCHEDULED YES NO YES NO YES NO YES NO YES NO YES NO 47
Slide 48: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 PHASE-IN CHECKLIST 36. Provide a call back roster to the KO. 37. Complete all work/tasks required under the Phase-In plan. 38. Provide the KO with a detailed final report outlining all work/tasks accomplished. 39. Perform a walkthrough with the Service Provider to document the work in process and the status of Center operations. 40. Perform a walkthrough of the facilities to identify any problems that may have occurred subsequent to the original inspection. 41. Fully operational Post Phase-In: (Phase III: To Begin at First Performance Period) 42. Receive/Conduct training within the first 90 days of the start of the first performance period. 43. Storage and Handling of Hazardous Material: (16 Hours) 44. First Responder Awareness Level; Service Provider provided 45. Hazard Communication; Service Provider provided 46. Hazard Reporting; Service Provider provided 1.3 Service Provider Surveillance Reports COMPLETED/CERT/ SCHEDULED YES NO YES NO The Government uses the Service Provider Surveillance Report (SR), see Figure 0-5, Service Provider Surveillance Report below, for day-to-day surveillance of Service Provider performance of contract APLs and Non-APL requirements. The Surveillance Specialists document nonconforming items/samples/processes in table field 24 of the SR and satisfactory performance and other additional nonconforming comments in cell 24 of the SR. Surveillance Specialists use on-the-floor direct observation, review of IFS or EMACS data elements, and the review of Service Provider documents and records to perform surveillance of the Service Provider. Surveillance Specialists also document surveillance results on the SR, recognizing both nonconforming as well as outstanding Service Provider performance and ensure the update of this information into the Government Scheduling Database and Surveillance Tracking Tool (see DDC Handbook for Managing DDJC Installation Services in a Post A-76 Environment, Appendix XX for a discussion of these tools). The Government’s dedication to achieve the most complete and objective recognition and documentation of the Service Provider’s compliance or non-compliance with the contract APL or Non-APL requirements governs the surveillance method or methods used. 48
Slide 49: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 SERVICE PROVIDER SURVEILLANCE REPORT 1. Solicitation #: Pre-Surveillance Information 4.Government Evaluator’s Name/Title (print): Service Provider Information 2. Service Provider Name: 3. Service Provider Acknowledgement Signature and Date (if required): 5. Evaluator’s Signature: 6. Surveillance Date and Time: 7. Primary Function Surveyed: 8. Area or Bldg #:  Management & Administration  Facilities Maintenance  Equipment Maintenance  Grounds Maintenance  Mail Services  Utilities Operation and Maintenance  IFS or EMACS  Reports/Records/Certifications 9. Reference: 10. Acceptable Performance Level (APL): 11: Specific Activity Surveyed: 12. Standard & Summary of Requirements: 13. Surveillance Methods: Surveillance Data & Results DOCUMENT OBSERVATIONS OF EXCEPTIONAL OR COMMENDABLE PERFORMANCE IN BLOCK 24 BELOW. 14. Lot 15. Sample 16. 17. 18. 19. 20. 21. 22. Total Non- 23. Surveillance Size: Size: Accept Reject Accept Reject Accept Reject Conforming Results: Critical Critical Major Major Minor Minor Samples:  Acceptable  Unacceptable 24. Comments:  Continued on Back of Form - Additional Observations, Notes, Comments Figure 0-5, Service Provider Surveillance Report 49
Slide 50: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.4 Terms and Definitions Definitions and acronyms for terms commonly used in the distribution process and referred to or discussed throughout the contract are contained in Section C-2 of the contract. The following list provides definitions for terms used in this document: 1.4.1 General Definitions Acceptable Performance Level (APL): The minimum performance of each requirement before the Government considers performance unsatisfactory. Acceptance: The act of an authorized representative of the Government by which the Government assumes for itself, or as an agent of another, ownership of existing and identified supplies, or approves specific services rendered, as partial or complete performance of the contract. Administrative Contracting Officer (ACO): Refers to a contracting officer who is administering contracts. “Contracting officer” means “Administrative contracting officer (ACO).” Audit: An examination of records or financial accounts to check their accuracy. Contract: A mutually binding legal relationship obligating the seller to furnish the services (including construction) and the buyer to pay for them. It includes all types of commitments that obligate the Government to an expenditure of appropriated funds and that, except as otherwise authorized, are in writing. In addition to bilateral instruments, contracts include (but are not limited to) awards and notices of awards; job orders or task letters issued under basic ordering agreements; letter contracts; orders, such as purchase orders, under which the contract becomes effective by written acceptance or performance; and bilateral contract modifications. Contracts do not include grants and cooperative agreements covered by 31 U.S.C. 6301, et seq. For discussion of various types of contracts, see FAR Part 16. Contract Quality Requirements: the technical requirements in the contract relating to the quality of the product or service and those contract clauses prescribing inspection, and other quality controls incumbent on the Service Provider, to assure that the product or service conforms to the contractual requirements. Contracting Officer (KO): A person with the authority to enter into, administer, and/or terminate contracts and make related determinations and findings. The term includes certain authorized representatives of the contracting officer acting within the limits of their authority as delegated by the contracting officer. Contracting Officer’s Representative (COR): An individual designated and authorized in writing by the KO to perform specific technical or administrative functions. Defect: see Nonconformance and Discrepancy Designee: The individual(s) designated and authorized in writing by the KO to perform specific technical or administrative functions. 50
Slide 51: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Facilities: Property, including plant equipment and real property, used for production, storage, maintenance, development, or testing. It does not include material, special test equipment, special tooling, or agency-peculiar property. Government Contract Quality Assurance: the various functions, including inspection, performed by the Government to determine whether a Service Provider has fulfilled the contract obligations pertaining to quality and quantity. Inspection: Examination of supplies and services (including, where appropriate, raw material, components and intermediate assemblies) to determine whether the supplies and services conform to contract requirements. See also “Monitor.” Inspection and Acceptance Clause: A term or condition used in contracts or in both solicitations and contracts, and applying after contract award or both before and after award that outlines the government’s rights to inspect and accept supplies and/or services. “In writing,” “writing,” or “written”: any worded or numbered expression that can be read, reproduced, and later communicated, and includes electronically transmitted and stored information. Lot: Any homogeneous group of items or services received or produced during a single event or production run. Examples of different kinds of lots include: XXXX. Monitor: To watch, keep track of, or check usually for a special purpose (see also inspection). Most Efficient Organization (MEO) Letter of Obligation (LoO): A formal agreement that an agency implements when a standard or streamlined competition results in agency performance (e.g., MEO). See OMB Circular A-76 for additional information regarding MEO LoO requirements. Modification: A minor change in the details of a provision or clause that is specifically authorized by the FAR and does not alter the substance of the provision or clause (see 52.104). “Must”: (see “shall”). Offer: A response to a solicitation that, if accepted, would bind the offeror to perform the resultant contract. Responses to invitations for bids (sealed bidding) are offers called “bids” or “sealed bids”; responses to requests for proposals (negotiation) are offers called “proposals”; however, responses to requests for quotations (simplified acquisition) are “quotations,” not offers. For unsolicited proposals, see FAR Subpart 15.6. Product Quality Audit: A quantitative assessment of conformance to required product characteristics. Process Quality Audit: An analysis of elements of a process and appraisal of completeness, correctness of conditions, and probable effectiveness. Provider: An agency, private sector, or public reimbursable source that is performing or will perform a commercial activity; sometimes referred to as a service provider (SP). 51
Slide 52: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Quality Audit: A systematic and independent examination and evaluation to determine whether quality activities and results comply with planned arrangements and whether these arrangements are implemented effectively and are suitable to achieve objectives. Quality Assurance (QA): The functions and associated actions performed by the Government to ensure that award requirements are performed in accordance with specified standards, and that an appropriate level of Service Provider Quality Control/Customer Satisfaction Plan activities are in place and operational. Quality Assurance Surveillance Plan (QASP: The government’s inspection plan. The QASP documents methods used to measure performance of the Service Provider against the requirements in the contract. The agency relies on the Service Provider to monitor daily performance using their own Quality Control/Customer Satisfaction Plan, but retains the right to inspect all services. When the agency makes a performance decision, the agency re-evaluates and modifies the existing QASP, based upon the selected provider and the selected provider’s accepted Quality Control/Customer Satisfaction Plan. Quality Control (QC/CSP): Those internal management functions that include, but are not limited to, training, documented procedures, inspections, and tests (taken at the point of performance) necessary to ensure that Service Provider products and services conform to contract requirements, specifications and standards. Quality Control/Customer Satisfaction Plan: As required by the solicitation, the Service Provider’s approach, staffing and procedures used by the prospective provider to meet the quality, quantity, timeliness, responsiveness, customer satisfaction and other service delivery requirements in the contract. Quality Control Plan: A self-inspection plan that is included in all offers and tenders. The quality control plan describes the internal staffing and procedures that the prospective provider will use to meet the quality, quantity, timeliness, responsiveness, customer satisfaction and other service delivery requirements in the contract. Quality Surveillance: The continuing monitoring and verification of the status of procedures, methods, conditions, products, processes, and services, and analysis of records in relation to stated references to ensure that requirements for quality are being met. Quality System Audit: A documented activity performed to verify, by examination and evaluation of objective evidence, that applicable elements of the quality system are suitable and have been developed, documented, and effectively implemented in accordance with specified requirements. Requests for Proposals: Requests for proposals (RFPs) are used in negotiated acquisitions to communicate Government requirements to prospective Service Providers and to solicit proposals. RFPs for competitive acquisitions shall, at a minimum, describe the— (1) Government’s requirement; (2) Anticipated terms and conditions that will apply to the contract: (i) The solicitation may authorize offerors to propose alternative terms and conditions, including the contract line item number (CLIN) structure; and 52
Slide 53: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 (ii) When alternative CLIN structures are permitted, the evaluation approach should consider the potential impact on other terms and conditions or the requirement (e.g., place of performance or payment and funding requirements) (see 15.206); (3) Information required to be in the offeror’s proposal; and (4) Factors and significant sub-factors that will be used to evaluate the proposal and their relative importance. “Shall”: Imperative. Mandatory. Obligatory. “Should”: An expected course of action or policy that is to be followed unless inappropriate for a particular circumstance. “Signature” or “signed”: The discrete, verifiable symbol of an individual that, when affixed to a writing with the knowledge and consent of the individual, indicates a present intention to authenticate the writing. This includes electronic symbols. Solicitation: Any request to submit offers or quotations to the Government. Solicitations under sealed bid procedures are called “invitations for bids.” Solicitations under negotiated procedures are called “requests for proposals.” Solicitations under simplified acquisition procedures may require submission of either a quotation or an offer. “Solicitation provision or provision”: A term or condition used only in solicitations and applying only before contract award. 1.4.2 DoD Dictionary The DoD Dictionary of definitions and terms is available on the Internet at http://www.dtic.mil/doctrine/jel/doddict. 1.5 Random Numbers In order to get a statistically valid result, randomly choose samples. Unfortunately, just picking a few out does not work. Experiments show that there are definite patterns to people’s choices, so it is not actually mathematically random. There are four options explored in this section to allow for the easy generation of random numbers. 1.5.1 Government Scheduling Database As indicated in Appendix XX of the DDC Handbook for Managing DDJC Installation Services in a Post A76 Environment, a random number generated has been added as a tool to help members of the Government quickly and accurately identify random numbers to use in their surveillance activities. Since the instructions for using this tool are specified in great detail in Appendix XX, no further discussion will be included here. 53
Slide 54: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.5.2 Web-based Random Number Generators Scientific researchers created a number of user-friendly web pages available (free of charge) to serve as sources of random numbers. The use of various, random methods produce the numbers. The detail exploration of one of the web pages follows, accompanied by web links to other sites. First, go to www.randomizer.org. Click the “Randomizer” button in the upper right hand corner. Now determine the number of samples required by using the Random Sampling Plan. Next, determine how many sets of numbers needed to generate (1) and how many numbers per set (2). The example shows three sets of five. The number range (3) determines the largest and smallest numbers possible; the example might generate numbers from 1 to 50, but never generates a 51 (or anything higher.) The default option for uniqueness (4) is “Yes,” which is what should be used. That way, one does not end up with any duplicates. 54
Slide 55: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 55
Slide 56: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Finally, click the ‘Randomize Now’ button(5) and the results will appear in a pop-up window. Other useable web sites include http://www.random.org/ or http://random.mat.sbg.ac.at/. A web search reveals even more. 1.5.3 Dice Another method generates random numbers by rolling 10-sided dice. The dice prove inexpensive and available through hobby shops, gaming supply stores, some bookstores, and the internet. First, determine the number of samples needed by using the Random Sampling Plan. 56
Slide 57: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Rolling one die for each digit desired will produce the necessary numbers (although you will have to omit any numbers outside your range and re-roll.) It is important to note that each die is one digit- do not add them together or it will produce an invalid result. For example, a roll produces the number 5. Rolling again gives an 8- and so the number is 58. To save time by using multiple dice, it is best to have dice of different colors- assign, for example, the red die as the first digit, the blue as the second and so on. 1.5.4 Random Number Table Determine the number of required samples by using the Random Sampling Plan (Figure 0-3, ). The following represent methods for selecting random numbers using a random-numbers table: a) Hang a page of random numbers on the wall and throw a dart at the page. The selected number represents the number hit by the dart. This method is valid; however, it drastically limits the life of the table. Place the page of random numbers on a table. Close your eyes and drop a sharpened pencil on the page. The number on which the pencil lands, becomes the selected number. Place the page of random numbers on a table. Close your eyes and drop your finger to the page. The number on which your finger lands, signifies the selected number. b) c) 1.5.4.1 Procedure Assume that the numeral 150 represents the lot size. The Random Sampling Scheme requires 20 samples. Using one of the random-number, selection methods listed above, perform the following: a) b) c) d) Select which page of random numbers to use. Use the table to select each of your samples. Predetermine which direction to move on the page, e.g., either up or down the column. Since only 20 samples are required, by using one of the methods listed above determine which two digits to use from the random numbers (first set or last set), e.g., if the random number is 05119, use either the first set “05” or the second set “19.” Again, determine the starting point on the page by one of the methods listed above. Are the two selected digits between 01 and 20? ♦ ♦ e) If yes, that is the first sample element. If not, move along the column in the predetermined direction until you find a number between 01 and 20. 57
Slide 58: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 f) g) Write it down. Continue to move along the column until you find the next number between 01 and 20. This is the second element. Repeat the process until all 20 samples selected. Figure 0-6, Uniform Random Numbers Table UNIFORM RANDOM NUMBERS 18545 50534 57654 25519 35477 59267 72742 53968 63679 54095 62828 19097 09877 32093 23518 33317 34192 06286 39824 74264 20510 09198 56256 04431 22753 28341 25428 08806 98858 04816 57395 86230 47495 13908 97015 01923 29260 32771 71002 58132 26970 37647 26282 89759 69034 22436 77006 87841 94322 45526 11295 38482 31143 00079 75787 69149 67519 57039 14117 42323 28104 81609 91290 07990 76185 94385 94109 03431 32204 22109 81862 08027 74076 38644 04778 73447 56850 35023 07253 48217 82190 63600 88844 70330 98476 18420 56884 96542 64281 38147 55022 85055 19947 55160 90685 47610 81177 63651 41351 80384 33376 53038 08365 00158 83780 35701 29289 73598 27279 59152 89522 90500 97889 74773 77107 96733 01217 59406 56629 55279 79551 50680 92058 31968 30333 34778 88681 71134 03782 34634 83104 61482 88847 98228 44814 93649 79883 10670 71436 73110 37112 16869 53023 00056 76250 88491 01335 83390 77594 68443 81744 61593 83630 28631 76632 79937 64199 36565 79004 02808 65591 53388 58898 92705 20219 27951 04166 92908 94963 72077 02243 74887 39511 18079 61379 37598 87381 95437 99133 36259 06026 27285 15336 23322 77333 01984 88335 83475 03859 82321 ROW 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 57245 42726 82768 97742 48332 26700 66156 64062 24713 90417 78886 39681 38265 34270 73869 21732 72583 72678 88572 82868 09949 41942 46777 58280 87042 72183 76768 21703 78269 49285 90016 63651 02780 49414 53861 78677 74550 22866 39666 58321 32694 58918 38634 40484 16407 10061 95591 18344 86557 54696 45196 41885 34748 36017 49811 49483 01294 59988 56572 91440 74928 17867 55080 39856 16066 64616 12087 16579 10303 93677 37186 09022 74046 55123 84403 64152 71309 56563 08654 01941 20944 16317 58225 58646 55281 38145 36929 34912 38823 77830 06246 55789 91140 27720 26415 21914 19059 29786 71134 93350 99430 74302 36354 70600 89308 95751 91955 10577 12812 88477 54047 59932 58670 34392 12212 09820 64815 95810 95319 94928 82255 69089 64853 86554 66839 18981 84543 05368 20952 73450 24571 14058 52233 48300 85172 81499 86187 05666 44251 06204 34292 53491 94179 91704 38436 21413 78027 83879 99675 19044 60057 09589 98911 86291 19894 26247 29515 05512 01956 63694 50837 42733 81250 74299 22119 58315 56808 68282 23446 66743 12818 96404 35273 85390 70717 55402 30896 71004 90092 95542 54907 59819 43892 86688 89926 60943 41864 27318 25225 97340 58
Slide 59: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 ROW 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 17601 20862 71490 21050 91632 46988 72200 88621 15384 46499 13520 24323 99090 97585 97622 24260 65250 45817 48898 23900 38635 58095 76474 03535 14364 91751 56151 57881 36126 73389 77383 62798 12283 78433 26348 26078 40115 81560 34074 42176 63288 88820 63532 32926 64222 83428 12825 96199 84607 24023 25822 03326 94631 96795 00280 55543 33311 85282 21646 83876 37470 28088 87304 66539 49005 40345 70021 82782 29817 59329 15295 70899 33283 40683 96248 00561 49762 29480 36705 70829 62666 51484 81350 62381 33240 20433 06120 96951 78903 28217 54191 55711 55848 78463 32091 17985 79714 73922 87734 68919 58594 75111 34806 73508 77723 91349 55139 59108 47744 61645 07176 90578 22926 43246 51669 66929 84063 60474 29955 41177 65497 83043 00654 77627 59356 05941 58461 78977 25690 UNIFORM RANDOM NUMBERS 27626 48687 42885 25858 19524 15866 52508 03763 81931 99510 77480 43874 09162 72191 20199 09369 66338 43447 80685 33189 83977 41560 08796 84200 81458 27541 56894 12315 45224 84268 53522 31800 66357 47103 79958 73444 45412 36149 05775 80949 34615 61592 12791 09123 20301 09754 13225 98928 09557 24345 86900 33049 26532 44976 00625 70046 19009 79836 00319 74261 49987 25002 90082 53719 73933 18981 42047 01608 35856 42903 65636 06834 13393 41724 15977 93311 31434 21358 21187 34520 32843 12888 12459 85668 63484 22365 16987 57138 41160 90207 37331 09987 00849 76307 15663 85945 64343 51848 44430 92899 70848 24081 39124 57613 94310 80187 35389 23002 05068 19651 86698 94414 46016 35965 68645 84216 62555 10263 84024 23194 47217 64714 19977 61255 19015 49481 95862 98248 19619 29671 95808 03971 91244 96668 38846 34697 72522 20566 58794 89290 75411 87196 79404 58350 93807 94363 26207 17199 47976 21910 55545 09851 06266 42758 41128 04327 78875 70694 16868 88935 03696 26059 40450 17597 40550 82785 60451 46995 90319 30877 33299 57383 53920 98033 38118 06520 65865 55961 83365 29613 79050 48921 06167 91840 59085 45363 67845 04060 55289 69917 85718 99586 41858 79711 87819 12237 04753 92983 27543 06677 16626 74346 77583 74445 35414 15064 25914 59892 92889 88073 26284 33116 82640 95592 68622 95764 87268 01065 81108 25164 83227 82930 46473 43837 22845 69090 39594 28471 46920 17771 94811 81226 58064 18019 96453 15382 42424 81523 43147 00023 61660 18723 77219 35623 84838 33522 54525 65998 11389 36394 66163 35041 71000 45273 39120 24888 38493 13359 84716 71167 36010 87266 98330 16069 63392 51177 15831 55264 29476 17621 74971 94760 58031 60124 18190 95000 66290 06861 98701 67008 34272 16894 15300 50106 11957 50803 18758 82770 53090 12869 49766 78409 79315 97848 18949 63511 35706 34765 02600 05321 25371 59
Slide 60: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 ROW 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 1.6 39033 46964 03528 67027 65462 55797 58697 97876 72443 96152 10277 77557 68784 67778 94104 70107 07847 33167 84527 68857 90715 92683 36641 73352 95254 56009 76551 02607 80526 64926 25011 85951 30262 06985 17900 64852 35411 88986 74705 01804 29740 74347 17392 84279 20438 19215 13183 62087 33378 86663 54232 16944 81837 53497 37719 27473 01665 UNIFORM RANDOM NUMBERS 91718 77485 32496 30737 14953 97658 71613 90353 31679 22941 92131 69021 54500 80074 94364 10164 09552 74361 46123 13020 88885 65569 96791 66118 06653 93978 51961 97609 87623 55326 85282 86292 06156 76680 62398 79369 12197 48335 97410 30976 36130 53674 71908 68837 13393 23547 59252 35488 99845 48666 77604 36266 18186 60757 17714 74666 68312 47577 42709 15927 34923 21688 59909 92158 59680 25487 39759 85954 48407 73072 92336 68769 00400 80433 30888 34977 28551 78189 21325 99309 63169 05817 97367 18328 77374 63535 97588 63841 00907 66565 18492 53236 17687 98213 59681 69056 03195 70930 66272 98318 17867 02795 55016 78292 23675 75873 14727 76237 94490 12242 23016 08916 93364 38520 95615 73795 19548 24925 91666 88254 04718 84126 41027 47358 31350 00484 56914 68039 34164 70860 01706 03219 57293 Random Sampling Plan 1.6.1.1 Normal Inspection LOT SIZE 2-8 9-15 16-25 26-50 51-90 91-150 151-280 281-500 501-1,200 1,201-3,200 3,201-10,000 10,001-35,000 35,001-150,000 150,001-500,000 500,001-Over NORMAL INSPECTION SAMPLE SIZE 2 3 5 8 13 20 32 50 80 125 200 315 500 800 1250 CRITICAL AQL 0.065 ACCEPT REJECT 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 1 2 1 2 2 3 MAJOR AQL 0.40 ACCEPT REJECT 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 1 2 2 3 3 4 5 6 7 8 10 11 MINOR AQL 6.5 ACCEPT REJECT 0 1 0 1 0 1 1 2 2 3 3 4 5 6 7 8 10 11 14 15 21 22 21 22 21 22 21 22 21 22 1.6.1.2 Tightened Inspection LOT SIZE 2-8 TIGHTENED INSPECTION SAMPLE SIZE 3 CRITICAL AQL 0.065 ACCEPT REJECT 0 1 60 MAJOR AQL 0.40 ACCEPT REJECT 0 1 MINOR 6.5 ACCEPT REJECT 0 1
Slide 61: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 9-15 16-25 26-50 51-90 91-150 151-280 281-500 501-1,200 1,201-3,200 3,201-10,000 10,001-35,000 35,001-150,000 150,001-500,000 500,001-Over 5 8 13 20 32 50 80 125 200 315 500 800 1250 2000 0 0 0 0 0 0 0 0 0 0 0 0 1 2 1 1 1 1 1 1 1 1 1 1 1 1 2 3 0 0 0 0 0 0 0 0 1 2 3 5 8 12 1 1 1 1 1 1 1 1 2 3 4 6 9 13 0 0 1 2 3 5 8 12 18 18 18 18 18 18 1 1 2 3 4 6 9 13 19 19 19 19 19 19 1.6.1.3 Reduced Inspection LOT SIZE 2-15 16-25 26-90 91-150 151-280 281-500 501-1,200 1,201-3,200 3,201-10,000 10,001-35,000 35,001-150,000 150,001-500,000 500,001-Over REDUCED INSPECTION SAMPLE SIZE 2 2 2 3 5 8 13 20 32 50 80 125 200 CRITICAL AQL 0.065 ACCEPT REJECT 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 MAJOR AQL 0.40 ACCEPT REJECT 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 2 1 3 1 4 2 5 MINOR 6.5 ACCEPT REJECT 0 1 0 1 0 1 0 2 1 3 1 4 2 5 3 6 5 8 7 10 10 13 10 13 10 13 1.7 Government Checklists TBD (Develop Installation Service Checklists) 1.7.1 Policies and Procedures POLICIES AND PROCEDURES CHECKLIST Note: Policies and procedures exist to ensure consistency and predictability within a process and the organization. The Government verifies compliance to policies and procedures by performing audits and surveillance. 1. Current policy or procedure posted 2. Policy or procedure signed and dated by authorizing personnel 3. Copies of obsolete policies and procedures purged 4. Employees possess easy access to the policies and procedures 5. Employees trained on the policy or procedure 6. Employees following the established policies and procedures YES NO 61
Slide 62: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 POLICIES AND PROCEDURES CHECKLIST Note: Policies and procedures exist to ensure consistency and predictability within a process and the organization. The Government verifies compliance to policies and procedures by performing audits and surveillance. 7. Policy or procedure adequate to accomplish its intended purpose 1.7.2 Security SECURITY CHECKLIST Note: The Government verifies the security of employees, the facility, and the materials in the depot from subversive activities and/or acts of internal security violations. 1. Service Provider reports security violations to the KO within one working day 2. All employees and visitors wear visible security (ID) tags 3. Controls established to recover the security badges upon completion of the visit 4. Visitors escorted by the Service Provider at all times 5. Service Provider personnel share PIN or Card Access Badges 6. Controlled inventory items properly identified, segregated, and stored in secure locations 7. Secured locations properly locked 8. Access to secured locations restricted to authorized personnel only 9. Secure location logs completely filled out and signed 10. Secure location logs reviewed and signed by supervisory personnel in a timely basis 11. Lights functioning in areas such as parking lots, ingress, and egress points 12. Key control effective 13. Evidence of pilfering or theft 14. Internal overheads, cages, vaults, warehouse doors, and other areas for reasons of internal security before leaving an area unattended and prior to the conclusion of any workday 15. Physical security planning performed to identify all security requirements applicable to all items received, stored, and issued 16. Physical Security Program developed and implemented to comply with all security requirements 17. Physical security facilities utilized, maintained, and controlled and necessary administrative controls established 18. Keys kept within the Key and Lock Control Program under continuous accountability at all times and the number of individuals authorized to draw keys limited to a minimum commensurate with security and operational requirements 19. Conforms to the IT standards established by DLA to maintain network integrity/security and verifies interoperability within DLA and with the DoD community 1.7.3 Safety SAFETY CHECKLIST Occupational Safety and Health Instruction: 1. Employees informed of the occupational safety and health program with emphasis on rights and responsibilities 2. Documented evidence that employees assigned to new positions or new tasks informed of the hazards of the new tasks prior to performing the tasks YES NO YES NO YES NO 62
Slide 63: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 SAFETY CHECKLIST 3. The KO received no later than the start of full performance a copy of Service Provider’s OSHA compliant safety plan that meets or exceeds all applicable requirements of the OSHA standard applicable to the hazard or task, including emergency actions and provided all updates and revisions to the safety plan within 30 calendar days of change 4. The KO received within 24 hours of any accidents and received a copy of any mishap report within six workdays IAW Section C-6.6, Mandatory Reports 5. Monthly copy of current Log of Work Related Injuries and Illnesses OSHA Form 300 submitted to the KO 1.7.4 Training TRAINING CHECKLIST Note: The Government audits training records to ensure complete and up-to-date information on the records. The audit also verifies that an employee performing a particular job function received required training. Training Records Audited: 1. Asbestos 2. Compressed Gas Cylinders 3. Confined Spaces 4. Environmental Protection Agency (EPA) Stratospheric Ozone Certification (MVAC) 5. Equipment Operation and Maintenance 6. First Responder Awareness Level 7. Hazard Communication 8. HAZMAT Transportation 9. HAZWOPER 10. Information Assurance 11. Lead Exposure 12. Oil Pollution/Oil Recycling 13. Ozone Depleting Substances 14. Pesticide Applicator Certification 15. Respirators 16. RCRA for Generators of HW 17. Security 18. Supply 2000 (S2K) 19. X-Ray Verification of Training: 20. Training records available 21. Training records complete with a) Name of employee trained b) Name of the course trained c) Description of the training c) Source of the training d) Date training completed by the employee 22. Employees working in a “training required” position fully trained YES NO YES NO 63
Slide 64: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 1.7.5 Documents and Records DOCUMENTS AND RECORDS CHECKLIST Note: Consider all records and documents as candidates for inclusion in an audit. This includes electronic and hardcopy documents. 1. Records accessible to authorized individuals 2. Records complete a) All blocks filled in. If a block does not apply, “–“(dash) or “N/A” inserted in the block b) All records dated with complete date not just day and month c) All signature blocks signed 3. Corrections made properly a) Word blackout or erasure b) Errors lined through with a single line c) The line-through initialed and dated by the individual making the correction d) The correct information inserted above or below the lined-through error 4. Any documents missing ( Note: If a document filled out daily, weekly, or monthly, the audit verifies the presence of all documents without a break in the chronology.) 5. Records protected from loss, damage, or unauthorized access and use 1.7.6 Government Furnished Equipment GFE MAINTENANCE CHECKLIST Note: The Government can perform record audits or observation audits, or a combination of records and observation audits to ensure compliance to equipment maintenance and care. 1. Daily, weekly, or monthly operator-maintenance check sheets completed 2. Maintenance work orders for GFE completed 3. GFE well-maintained (determine through observation) Lights, horn and other safety devices working properly a) No major leaks of oil or hydraulic fluid b) Fork safety pins present and functional 4. Evidence of equipment cannibalization 5. Observe start-up of equipment: a) Did the operator perform pre-startup checks b) Did the operator complete and sign the pre-startup check sheet 6. Is the equipment operated in a safe manor 7. No excessive speeding 8. Use of safety equipment such as safety harness and lanyard 9. No standing on forks or open pallets on the lift 1.7.7 Customer Satisfaction CUSTOMER SATISFACTION CHECKLIST Note: The Government audits the customer satisfaction program. Regular surveillance of the customer satisfaction program measures program effectiveness as a tool used to improve customer communication and satisfaction. 1. Documented evidence of Service Provider communication with the customer dealing with improved customer service and satisfaction YES YES YES NO NO NO 64
Slide 65: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 CUSTOMER SATISFACTION CHECKLIST Note: The Government audits the customer satisfaction program. Regular surveillance of the customer satisfaction program measures program effectiveness as a tool used to improve customer communication and satisfaction. 2. Documentation indicates the communication occurs on regular bases 3. Documented evidence of customer-complaint investigations 4. Corrective actions taken place to improve customer satisfaction and rectify customer complaints 5. Evidence of analysis of customer comments, complaints, and suggestions 6. Report issued summarizing the results of analysis of customer issues and disseminated to the workforce 7. Service Provider’s employees informed of customer issues 8. Employee teams organized to work on improving customer response and customer satisfaction 1.8 User Complaint Record USER COMPLAINT RECORD Name: Date Received: COMPLAINT SOURCE INFORMATION Title: Organization: Office Location: Time Received: Telephone #: YES NO Government Representative Receiving Information: COMPLAINT INFORMATION Describe the situation and events: Contract Reference including Requirements: Government Actions and Response (include the Surveillance Log reference # used to document the investigation): Service Provider Actions and Response: SERVICE PROVIDER INFORMED Title: Telephone Date: #: Title: Telephone #: Date: Person Originally Informed: Person Informed of Investigation Results: Time: Time: Response or action Reported by the Service Provider (attach supporting documentation provided by the Service 65
Slide 66: DDJC Installation Services DRAFT 1 – Quality Assurance Surveillance Plan 11/25/11 Provider): PREPARER’S INFORMATION Telephone #: Name: Title: Completion Date: 66
Slide 67: 1.9 Sampling Tally Checklist P.O. #: P.O. #: Item#: Service Provider: Inspection Time Sample Qty.: Surveillance Specialists: Date: List inspection criteria in this column such as: 9:00 315 9:45 315 10:15 315 Total Tally each row of nonconformances in this column. List in these columns quantity of nonconformances found. Wrong color Torn Case count wrong 0 1 3 0 0 0 2 0 1 2 1 4 Tally columns Grand total of nonconformances in this box. Total Nonconformances #’s Accept / Reject 4 0 3 13/14 13/14 13/14 INSPECTION LOG – List and explain any problems found during planned and random inspections. 7 67
Slide 68: 1.10 Service Provider Shall Requirements 68