文档文库
手机版
投诉建议
热门搜索： 心得体会 演讲稿 思想汇报
首页 > how do we measure it

how do we measure it

发布时间：   来源：文档文库   
小 中 大
字号：
手机查看
NationalResearchCouncilCanadaInstitutefor
InformationTechnology
Conseilnational
derecherchesCanadaInstitutdetechnologiedel’information
Ifsoftwarequalityisaperception,howdowemeasureit?
W.M.Gentleman
SoftwareEngineeringLaboratoryJuly1996
NRCNo.40149

ThispaperwasoriginallypublishedinTheQualityofNumericalSoftware:AssessmentandEnhancement,RonaldBoisvert,ed.theProceedingsofIFIPWG2.5WorkingConference7,Oxford,UK,7-12July1996,Chapman&Hall,London,pp.32-43.
ThispaperwasalsopublishedinSoftwareQualityWorld-wide:WhatArethePracticesinaChangingEnvironment,ProceedingoftheSixthInternationalConferenceonSoftwareQuality(6ICSQ,Ottawa,Canada,28-30October,1996,pp.335-345.
Copyright1996by
NationalResearchCouncilofCanadaPermissionisgrantedtoquoteshortexcerptsandtoreproducefiguresandtablesfromthisreport,providedthatthesourceofsuchmaterialisfullyacknowledged.
Additionalcopiesareavailablefreeofchargefrom:
CommunicationsOffice
InstituteforInformationTechnologyNationalResearchCouncilofCanadaOttawa,Ontario,CanadaK1A0R6
Copyright1996par
ConseilnationalderecherchesduCanada
Ilestpermisdeciterdecourtsextraitsetde
reproduiredesfiguresoutableauxduprésentrapport,àconditiond’enidentifierclairementlasource.
Desexemplairessupplémentairespeuventêtreobtenusgratuitementàl’addressesuivante:Bureaudescommunications
Institutdetechnologiedel’informationConseilnationalderecherchesduCanadaOttawa(OntarioCanadaK1A0R6
ii

Ifsoftwarequalityisaperception,howdowemeasureit?
W.M.Gentleman,
NationalResearchCouncilofCanada,Ottawa,Canada,K1A0R6
Tel.(613993-3857,Fax.(613952-7151,gentleman@iit.nrc.ca
Abstract
Forovertwentyyears,metricshavebeenbeinginventedtomeasuresoftwarequality.Andyetquantifyingqualitypresupposesagreementonwhatconstitutesquality.Qualityhasbeenportrayedasanabsolutequantity,subjecttoobjectivemeasurements.Webelievethisefforthasbeenmisguided.Wearguethatquality,likebeauty,isintheeyeofthebeholder–thatis,thatqualityisnotabsolute,butdependsontheperspectivetakenbytheevaluator.Assuch,anydirectmeasureofqualitymustnecessarilybesubjective,summarizingtheimpressionsofsomeparticularclassofpeoplewhointeractwiththeproduct.Indirectmeasuresofqualityarelessobjectivethantheymayappeartobe–beyondthearbitrarinessofthechoiceofmeasure,andanydifficultyinitsinterpretation,thereisalwaysthetenuouslinkofthemetrictotheperceptionofqualitybyanyspecificgroup.Theneedforthisnovelpointofviewisespeciallyclearlyillustratedbymathematicalsoftware.
Keywords
Softwarequality,measurement,assessment,subjective,objective

NRCnumber:
40149
1

INTRODUCTION
Atarecentmeeting,twoeminentcomputerscientistswerediscussingthenumericalcomputingenvironmentMatlab.‘CouldMatlabbespace-worthy?’askedthefirst.‘Nevermindwhetheryoucouldflyitinaspaceraft’,respondedthesecond,‘wouldyouevendependontheresultsofitscalculationsinaspaceflight?’TheimpliedcriticismwasthatMatlabhadnotbeenbuiltthroughtheprocessofformalspecification,traceabilitytorequirements,formalreviews,andtestingagainstspecificationwhichistypicallydemandedofsafetycriticalsoftware.Mostnumericalanalystswoulddisagreewiththiscriticism.Asacommercialproduct,Matlabhasbeenimplementedbyknowledgeableandskilledexperts,basedonwellstudiedalgorithms.Beyondtestingbythedeveloper,andwidespreadgeneraluse,ithasbeenintenselyusedby,andindeedbeenthesubjectofresearchby,otherexpertsinthearea.Weaknesseshavebeenidentified,buttheconsensusseemstobethatthebaseproductissolid.Hastheconventionalprocessproducedsimilarendorsementfromtheusersofspacesoftware?Tociteafewincidents:theFebruary1991Patriotmissilefailureillustratesthatusersmayoverlookspecifiedlimitations;theMay1992failureoftheshuttleEndeavourtorecoverasatelliteautomaticallyillustratesthatthestandarddevelopmentprocessdoesnotprotecttheunwaryagainstwellknownanomaliesoffloatingpoint;andtheJune1996Ariane5failureillustratesshortcomingswithblindlybelievingspecifications.Inthepast,muchofthethinkingaboutqualityhasbeeninthecontextofone-on-onecustomer/contractorrelationship.Galsworthy(1912extolledthevirtuesofhandcraftedcustomproducts.Today’scustomeroftenfacesaverydifferentsituation:achoicebetweencompetitiveoff-the-shelfproducts.(Notethattheuserofnumericalsubroutinelibraries,orevenofpre-existingnumericalmethods,hasalwaysbeeninthissituation.Eachoftheseproductshasitsownspecification,andnotonlyarethesedifferent,notonlyhadtheindividualcustomernopartincreatingthatspecification,buttypicallythefullspecificationisnotaccessibletohim.Noneofthespecificationsislikelytomatchanyindividualcustomer’sneedsexactly,soelicitingdetailedrequirementsfromhimintheabsenceofknowledgeabouttheavailableproductsissimplyanexerciseinraisingfrustrationlevels.Forthesereasons,‘correctimplementation’,inthesensethattheproductsconformtotheirseparatespecifications,islikelytobeaminorissueforthecustomer.Fieldexperienceislikelytobeofmorevaluethanformalspecificationandverification.
Quantifyingsoftwarequalityisimportantbecause,apartfromaestheticappreciationofqualityproducts,ourpurposeinexaminingqualityistofacilitatedecisionmaking.Oneexampleofthesedecisionsisthechoiceofproducts,whereseveralwouldappeartodothejob.Anotherexampleisthedecisionofwhethertoaccept,andtopayfor,aproductthatclaimstomeetaparticularneed.Weareoftenconcernedwithqualityandprice,e.g.whatqualityisavailableforagivenprice,orhowmuchextrawouldbetterqualitycost.Consequentlyanotherexampleisindecidingwhatinvestmentisworthmakinginordertoimprovethequalityofagivenproduct.Inmanycaseswhatwereallywanttodoistopredictwhatourownlevelofsatisfactionwiththesoftwarewillbe,beforewehavehadthechancetoexercisethesoftwareextensivelyinaparticularcontext.Thiscomesupbothwhenthesoftwareisunfamiliartous,andwhenitisnotyetcomplete.The
2

classicalqualityassurancemotivation(e.g.IEEE,1988ofmonitoringtheproductionprocessisonlypartofthestory.
Animportantconsequenceofqualityisthatitengenderstrust.Theuserfeels‘IfthedevelopertookpropercareofthedetailsthatIrecognizetheneedfor,theprospectisgoodthathealsotookpropercareofthatwhichIwouldcareaboutifIhadthoughtofit.’Thedetailsonwhichqualityisassessedandtrustisestablishedmaynotevenbeonesrelevanttothisapplication.‘Forme,IwouldsaysoftwarequalityoccurswhenIperceivethattheproducerappearstohaveappropriatelyaddressedtheissuesthatconfrontmeasauser.Thismeansmostlyanabsenceofevidencethatheorsheisignorantofsomethingthatreallymatterstomebutalsopositiveassurancesthatmyconcernsarereflectedinthesystem’(Johnson,1996.Usersmustmakedecisionsaboutwordprocessorsandspreadsheets,forinstance,andtheyeasilyidentifyquality(or,moresignificantly,lackofqualityinsuchproducts,withouteverformalizingneeds.Intheworldofmathematicalsoftware,oneofthemostvaluablecharacteristicstomanyusersisexactlywhenthesoftwaretakescareoftroublesomebutraresituations,sothattheuserneednotevenbeawarethatsuchsituationsmightexist–andcertainlywouldnotbeabletoenumerateandcharacterizethem.
Fornumericalsoftware,manypeopleusedtothinkperformance(speed,intermediatestore,pagingbehaviour,etc.wasallthatmatteredforquality,butaswidelyavailablemachinesbecomemorecapable,thisviewisfadingasithasforothersoftware(Carrol,1984.
Incommonparlancequalityisoftenassociatedwithdurabilityandlastingvalue.Forsoftware,sinceitdoesnotwearout,thispertainstoongoingneedforthesoftwareandtoresilienceofthesoftwaretochangesintheenvironment.
Thedefinitionofthetermqualityisanissue.AninterestingdiscussionofthemeaningofqualitycanbefoundinKitchenham(1986.Asurprisingnumberofpeoplestillthinksoftwarequalityissimplytheabsenceoferrors.Dictionarydefinitionsaretoovaguetobeofmuchhelp.TheonlyrelevantdefinitionofferedbytheOxfordEnglishDictionary(Oxford,1993,forinstance,ispeculiarexcellenceorsuperiority.Noteworthyhereisthatqualitycannotbediscussedforsomethinginisolation:comparisonisintrinsic.Manysoftwareengineeringreferences(e.g.Gilbert,1983;Schach,1990;Hailstone,1991;Tinnirello,1995definesoftwarequalityascorrectimplementationofthespecification.Suchadefinitioncanbeusedduringproductdevelopment,butitisinadequateforfacilitatingcomparisonsbetweenproducts.Standardsorganizationshavetendedtorefertomeetingneedsorexpectations,e.g.theISOstandardISO8492:1986(ISO,1986definesqualityasthetotalityoffeaturesandcharacteristicsofaproductorservicethatbearsonitsabilitytosatisfystatedorimpliedneeds,adding(Note–Inacontractualenvironment,needsarespecified,whereasinotherenvironments,impliedneedsshouldbeidentifiedanddefined.IEEEStd610.12–1990(IEEE,1990definesqualityas(1Thedegreetowhichasystem,component,orprocessmeetsspecifiedrequirements.(2Thedegreetowhichasystem,component,orprocessmeetscustomeroruserneedsorexpectations.AnolderIEEEdefinition,IEEEStdP1061–1988(IEEE,1988definesSoftwarequalityisthedegreetowhichsoftwarepossessesadesiredcombinationofattributes.
3

Softwarequalityisoftendefinedintermsofthefitnessoftheproductforitspurpose.Howeverdifferentpeoplehavedifferentpurposesforthesamesoftware.Anovicecasualuserisprobablymoreconcernedabouteaseoflearning,andaboutrobustnessagainstmisuse,thanaboutefficiency.Asystemintegrator,planningtoincorporatethesoftwareinsomelargersystem,mightbemoreconcernedaboutfailuredetectionandrecoverythanabouteaseofinitialinstallation.Athirdpartymaintenanceorganizationisconcernedwithissuessuchasinternaldocumentationandadequacyofscaffolding(e.g.testharnesses,testgenerators,andinstrumentationthatgobeyondissuesofdirectconcerntotheusers.Theseshowthatsoftwarequalityisnotabsolute,butisaperceptiondependinguponforwhomthequalityisevaluated.Moreover,softwarequalityismultifaceted,andtheimportanceofthedifferentfacetschangeswiththecontext,evenforthesamepersonatdifferentpointsintime.
Considerthepurposesofmathematicalsoftwareproducts,suchasnumericallibraries(NAGorIMSL,anumericalcomputationandvisualizationenvironment(MATLAB,asymbolicmathematicssystem(MapleorMathematica,oraframeworkforcomputationonspecifickindsofproblems(anoceanmodel,abombcode.Foranyoftheseproducts,inadditiontothesupportactivitiesfortheproducttherearepeopleusingtheproductforattheveryleastthreedifferentpurposes:1productioncomputationofresultsneededinotherdisciplines,2teachingstudentsaboutthemathematics,3researchintodevelopingnewmathematicalmethods.Theneedsofthesegroupsareoftennotjustdifferent,butconflicting–whatonegroupwouldregardasqualityanothermayregardasmakingtheproductunusable.Theoptimizationsthatmakethecodefastenough,andtheintermediatestoragecompactenough,forproductioncomputationtobepracticalmaymeanthecodeistoocomplexforstudentstolearnfrom,andthatinsightspossiblefromintermediateresultsandauxiliarycalculationsarenotavailable.Themathematicalrigournecessarytoprovethatnewalgorithmsforsymboliccomputationhavetakenallpossibilitiesintoaccountmaybesoclumsyastomakethesystemuselessforanengineerdoingexploratoryderivationsofformulaethatwouldonlybeusefultoprovideinsightiftheresultsaresimpleenough.Flexibilityprovidedbyfacilitatingchangestothesourcecodemaybeanecessityforuserstryingtodocomputationsbeyondthemodelthataframeworkdirectlyprovides,butitisanightmareforsupportpersonnelrespondingtoproblemreports,whohavenoeasywayofrecognizingwhethertheproblemwascausedbyadefectintheproductorbyauserchangebreakingsomething.(Evenrestrictinguserextensibilitytoplug-inmodulesstillleavesthisproblembecauseaninadequateAPIspecificationcanleadtosubtlefailuresoftheplug-in.
QUALITYATTRIBUTES
Aparticularlyimportantdistinctionisbetweenwhatrepresentsqualityfortheuserandwhatrepresentsqualityforthesupplierofacommercialproduct.Listsofattributesthatqualitysoftwaremustaddresshavebeensuggestedforsometime(e.g.Gilb,1977.Acuriousaspectoftheselists,explicitintheISO/IEC9126standard(ISO/IEC,1991,isthattheytypicallyonlyconsiderattributesofdirectsignificancetotheuser.(Thisstandardpointsoutthatthereareseveralpotentialviewsofquality,includingtheuser’sview,thedeveloper’sview,andthe
4

manager’sview,butonlytheuser’sviewisinthecurrentversion,withotherspromisedinlaterrevisions.Seealso(ISO,1987e.Suchalist,forinstance,mightbe:
UserVisibleAspects
••••••
Appropriatefunctionality
CoexistenceandinteroperabilityEaseofuse
Lackofsurprises
Adequateandusabledocumentation
Easeofinstallationandupdate/cutover,includingdata.
Alistofattributesofimportancetoasupplierwouldincludethesebecausekeepinguserssatisfiedisessential,butwouldgomuchfurther:
SupplierVisibleAspects
••••••••••
EaseoflearningformaintainersEaseofadaptability
Structurefortimelinessandcost-effectiveimplementationAdequacyofexceptionhandling
TestabilityandmeasurabilityofproductAnalyzabilityandpredictabilityofproductAdequacyofscaffoldingandsupporttoolsProfessionalismofprogrammingEfficiencyandperformance
Abilitytoconvincethirdpartiesofcorrectness,conformance,etc.?
MEASUREMENTOFPERCEPTIONS
Thedirectapproachtomeasuringqualitythenistostudytheperceptionsthatothershaveformedofthesoftware,andextrapolatethattooursituation.Acentralissuetoaddressiswhoseopinionswewant.Inpromotingsubjectiveassessmentofsoftwareproducts,weareinnosensesuggestingthatevaluationofsoftwarequalityshouldbelefttotheintuitionofthedevelopers.Oneobviouspossibilityisthatofexpertsinthearea.Thisis,effectively,whattherefereeingprocessofjournalsprovides,althoughthefactthatrefereesreportsarenotmadepublicdiminishesthebenefitthirdpartiescangainfromthem.Mostjournalswouldwelcomepapersthatarecritiquesandcomparisons,butthesearequiterare.CommercialpublisherssuchasOvumproducereportslikethisonpopulartopics,buttheytypicallyhavelimitedaccessibilityduetoprice.Anadvantageofexpertassessmentisthatexpertsarecompetenttoknowwhatarethepotentialstrengthsandweaknessestolookfor,andhowtostudythem.Thedisadvantageisthat
5

byknowingtoomuch,theexpertmaynotrecognizeobstaclesthatwouldimpedetheuseofthesoftwarebynovices,orbyuserscomingfromotherdisciplines.
Anotherpossiblesourceofopinionsisjournalisticreviewers,suchasthosepublishedinthecomputerpress.Anadvantageofthisgroupisthattheyhaveprofessionalincentivetodomanyreviews,andhencehaveabroadcontextagainstwhichtocompareproducts.Theyalsobecomeadeptatexplainingtheirimpressionsofaproducttoalargelynontechnicalaudience.Thishowevercanalsorepresentadisadvantageifareaderhasadeeperunderstandingoftheareathantherevieweraddresses.Anotherdisadvantageisthatjournalisticreviewerscanbebiasedbytheirpersonalneedsandexperience,whichareprimarilyjournalisticnottechnical.
Yetanotherpossibility,madepracticallargelybythenewsgroupsontheInterNet,istoassimilatetheexperiencesof‘userslikeme.’Newsgroupssuchascomp.soft-sys.matlab,comp.soft-sys.math.mathematica,andsci.math.symboliccontainmanyitemsthatprovideinsightintousers’impressionsofthatparticularsoftwareproduct:problemsusershaveinappreciatinghowtousethesoftware,creativewaysthesoftwarecanbeusedtoperformcomplextasks,desirableenhancements,etc.Itemswhereaquestionposeddrawsresponsesfromoneormoreotherusers,orevenfromsupplier’srepresentatives,areparticularlyinterestingtothirdparties.Distillingqualityevaluationsfromnewsgroupsisarduous,however,becauseofthefloodofitems,becausetheinformationisnotpresentedinaformsuitableforautomaticprocessing,andbecausethecontributorsdonotrepresentarandomsample(andindeeddonotexplicitlycharacterizethemselvesastobackground,sophisticationetc..Thesituationcouldbeimprovedif,ratherthananewsgroup,aWWWwebsitewasusedtocollectsuchitemsinadatabase.AlthoughHTMListooweaktofacilitatemathematicalnotationdirectly,diagramsandevenmathematicalnotationcanberenderedbyadroituseofGIF,whichformathematicalsoftwarewouldbeofrealbenefitoverjustusingplainASCIItext.Whileavendor’ssummaryofsuchuserfeedbackmightbedismissed,therawmaterialbeingavailablemeansothers,suchasausergrouporevenanindividualpotentialuser,canprovidetheirownanalysis.Analysisofsuchdatahasmuchincommonwithretrospectivestatisticalsurveys,andwhilesuchstudiesdonotenjoytheopportunitiesthatprospectivesurveyshavetouserandomizationtoeliminatebias,thereisasubstantialliteratureonhowtodetectandameliorate,ifnotcorrectfor,itseffects(Cochran,1963;Stephan,1958;Clark,1991.Vendorsmightinitiallybesensitivetonegativeimagepossiblefromsomeofthepostings,butthenewsgroupscontainthosenow,andonthewholeareofnetbenefittotheproducts.
Wearenotsuggestingthattraditional‘objective’metricsbeabandoned,butonlythattheyshouldbeappreciatedinadifferentlight.Iftheactualintentistomeasureperception,measurement,howeverobjective,ofattributesofthesoftwarethatwesuspectmightinfluenceourperception,isanindirectapproach.Relatingthesemeasuredattributestomeasuredusersatisfactionissurprisinglyrare(Buckley,1995.Thisindirectapproachappearstohavetheattractionofbeingmorequantitativeandprecisethanthediscursivepresentationwithcheckliststhattypifythedirectapproach.Italsoappearstobelessinfluencedbyindividualintuitionandtaste.Thereare,however,deeperconsiderationsthatneedtobetakenintoaccount:
6

•••••••••••
Thesetofattributestoconsiderisproblematical.
Theappearanceofobjectivityissomewhatmisleading,inthatmanyoftheattributesareinfactqualitative,notquantitative.
Evennotionallyquantitativeattributessuchasportabilityarenotsoinpractice.
Forsomeattributes,thearbitrarinessofanymetricmeansnumericalscorescandistortthepictureratherthanrefineit.
Theoremsprovedaboutthesoftwareobviouslyincreaseourunderstandingofit,butmaybeoflimitedapplicabilityunlesstherelevantconditionsofthetheoremcanbereadilyestablishedorthetheoremcanbeshowntoberobust.
Ostensiblyreproduciblecomputationalexperimentscanbecarriedoutbythevendor,byanindependenttestinglaboratory,orevenbythepotentialuser,tostudywelldefinedattributessuchasaccuracy,storagerequirementsorspeed,yetbatteriesoftestshavesimilarprovisos.Thereisalsothequestionastowhetherproblemsstudiedbycomputationalexperimentsshouldberepresentativeofproblemsinthearea,orshouldbeillustrativeofspecificstrengths(orweaknessesofthesoftware.
Experimentalassessmentofrepresentativetasks,suchasextendingthealgorithmorintegratingthesoftwareintoalargersystem,requirescontrolonsomanyfactorsthattheresultsareusuallybestunderstoodasanecdotal.
Someattributesareintrinsicallydifficulttoobserve,whichoftenleadstoinsteadstudyingsurrogatesthathopefullyexhibitsimilarbehaviour.
Atrapcompetitivedevelopersoftenfallintoismistakingascoreonthesurrogatefortherealobjective.
Interestingly,whenourrealpurposeistopredictperception,itmaynotbenecessaryfortheobservedmeasurementtohavearecognizedcausalrelationshipwithqualityatall,providedthatthereisanobservedstatisticalcorrelation.
Anevenmoreindirectapproach,currentlyinvogue,istostudytheprocessbywhichthesoftwarewasbuilt.Forcustomsoftwarethathasnotyetbeendelivered,andforwhichthereishencenousercommunity,itisnotobviouswhatelsecanbedone.TheISO9000standardfamily(ISO,1987a;ISO,1987b;ISO,1987c;ISO,1987d;ISO,1987e;ISO,1991basicallyrequiresthatwhateverprocessischosenfordevelopmentshouldbeunderstoodanddocumentedandshouldbemonitoredtoensureitisactuallyused.TheSoftwareEngineeringInstitute’sCapabilityMaturityModel,CMM,goesfurtherinrequiringwhatkeyareasprocessshouldaddress,andsuggestingthattheprocessshouldbemeasuredsoitcouldbeoptimized.Unfortunately,despitetheconsiderablepublicityoverthepastfewyearsforprocessimprovement,anddespitetheselfevidentnatureoftheassertionthatprocessmustmakeadifference,thereisadearthofquantifiedexperimentalscientificevidencethatcurrentmethodologieshavetheclaimedeffects.
Insummary,aqualityassessmentmustbepresentedinsuchawaythatinusingtheassessmentyourownjudgmentcanbeapplied.Anappreciationofhowthesoftwareisintendedtobeusedmaybeimportantinassessingitsquality.Anappreciationofwhowillbeusingitandhowtheywilluseitcertainlyis.
7

ASOUNDPRECEDENT
Scienceisconservativeandpreferstofollowsoundprecedents.Aninterestingprecedenttoourpointofviewtookplaceadecadeagointhefieldofacoustics,specifically,intheevaluationofloudspeakerquality.Atthetime,therewascontroversyaboutwhatweretheappropriatemetricstouse(Toole,1986a.Over50years,technicalmeasurementtechniqueshadbeendevelopedforqualityattributessuchasamplituderesponseasafunctionoflisteningperspective,andsuchasphaseresponse,transients,nonlineardistortions,andaudibilityofanomaliesinamplitude-responseandtime-domain.Theinterpretationofthesemeasuredresultsisnotobvious,astheresultsaretypicallynotsimplescalarsbutcurves,and‘evidencethatoffendstheeyemayormaynotindicatethepresenceofaproblemthatisoffensivetotheear’(Toole,1986a.Claimsofrelationshiptobetterqualitysoundweremutuallyinconsistent:forinstanceaflataxialamplituderesponseacrossfrequencymathematicallycannotcorrespondtoaflatfrequencyresponsewithrespecttototalradiatedacousticpoweroutputandviceversa,yeteachhadproponentsclaimingsuperiority.
Ofcourselisteningtestsandsubjectivepreferenceshavealwaysbeenusedforevaluatingaudioequipment.However,therewasawidespreadbeliefthatdifferencesbetweenlistenertastesweresolarge,nevermindthedifferencesinducedbydifferentlisteningconditions,thattherewasrealdifficultyindefiningthescopeandvalidityofanyresult.
Tooledevelopedamethodologyforsubjectiveexperimentswiththoroughcontroloftheacoustical,psychological,andexperimentalsourcesofvariability(Toole,1985.Using42listenerswithabackgroundofseriouscriticallistening(fromprofessionalsoundengineerstoaudiophiles,manyofthemmusicianshefoundthatincomparativeassessmentofloudspeakerqualitythevariationinsubjectivepreferenceratingsforthosewithnormalhearingwaslessthan0.5onascaleof0to10.Fromtheseprecisesubjectivemeasurements,hewasabletogobacktothetechnicalmeasurements(Toole1986bandshowwhichwereconsistentwiththesubjectivemeasurements,whichwereinconsistent,whereconventionaldataprocessingdiscardedsignificantinformationfromthedata,andwherenewtypesoftechnicalmeasurementsmightbeneeded.Recognitionthatperceptionwasthefundamentalcriterionofqualitythusnotonlyledtobettermethodologyformeasuringit,butalsoimprovedthetechnologyfortechnicalmeasurements.Incidentally,thisallledtosignificantimprovementsincommercialproducts.Thelongtermobjectiveoftheprojectwasachieved,i.e.todefineasetoftechnicalmeasurementsandtheformoftheirpresentationsuchthatinterpretationsbyexperiencedunbiasedobserverscorrespondstotheresultsofcontrolledlisteningtestsusingunbiasedlisteners.Becausedesignersfinditeasiertorelateaspectsoftheirdesignstoobjectivetechnicalmeasurementsthantosubjectivemeasurements,theimprovedtechnicalmeasurementsandtheabilityfromthemtopredictsubjectiveinterpretationledtoanewgenerationofloudspeakerdesignsperceivedtohavesignificantlyhigherqualityaudio.
8

PRODUCTREVIEWS
Althoughtheideaofmeasuringqualitybysurveyingusersisnovelinsoftwareengineering,itisnotatallunusualinreviewingotherproducts.Reviewsarebyexperts,whorelyfortheircredibilityprimarilyonthereputationoftheorganization,notontheirpersonalqualifications.Suchreviewsarepublishedformanykindsofproducts:audioequipment,cameras,automobiles,andpersonalcomputersoftware,tonameafew.Wewillconsiderbrieflyreviewsofautomobiles.Itisrecognizedthattheperceptionofqualityissodifferentforautomobileenthusiastsandforthoseconcernedonlywithsafe,reliable,andeconomictransportationthatnotonlyarethereviewsquitedifferent,buttheyarepublishedindifferentplaces:RoadandTrackandMotor,forinstance,insteadofConsumersReportsandWhich?Itislesswellknownthatthereisyetanothersetofreviews,publishedinthetradepressoftheservicestationbusiness,consideringthemechanicsperceptionsofdifficultiesandtricksfordealingwithspecificmodels.
Reviewsforallthesegroupsincludespecificlabmeasurements,suchasturningradius,brakingdistanceorfuelconsumption,butthedominantpartsofthereviewsaretheroadtestsandlongtermownerreportsproducedbyexpertdrivers.Aparticularlyvaluableserviceprovidedbysomereviews,suchasConsumerReportsAnnualAutoIssue(Consumers,1996,isthecompilationofownersurveysofreliabilityandotherexperienceinuseofalternativecurrentandpastproducts.Thiskindofproductreviewwouldbeequallyapplicable,andequallyvaluable,formathematicalsoftware.
CONCLUSIONS
Wehavearguedthatmeasuringqualityisnotjustforqualityassurance.Wehavesuggestedthatitiswisetobreakfreefromnarrownotionsofwhatconstitutesquality.Fromauser’sperspective,wehaveindicatedtheimportanceofconsiderationofcompetitionandofsoftwarelifetimeovermultiplereleasecycles.Wehaveassertedthatsubjectiveassessmentofqualitycanbeuseful,andthatobjectivemeasuresshouldbeusedtosupportsubjectiveassessment.
REFERENCES
Buckley,M.andChillarege,R.(1995DiscoveringRelationshipsBetweenCoverageand
CustomerSatisfaction.1995ConferenceonSoftwareMaintenance,October17–20,Opio,France,192–201.
Carrol,J.M.andRosson,M.B.(1984BeyondMIPS*:PerformanceIsNotQuality.BYTE,9(2,
February,168–72.
Clark,K.B.,andFujimoto,T.(1991ProductDevelopmentPerformance.HarvardBusiness
SchoolPress,Boston
Cochran,W.G.(1953,1963SamplingTechniques.JohnWileyandSons,NewYorkConsumersReports(1966AnnualAutoIssue,61(4,April.
9

Galsworthy,John.(1912TheInnofTranquillityandOtherEssaysandStudies,Charles
Scribner’sSons,NewYork
Gilb,T.(1977SoftwareMetrics,Winthrop,Cambridge,MA.
Gilbert,P.(1983SoftwareDesignandDevelopment.ScienceResearchAssociates,Inc.,
Chicago.
IEEEStdP1061–1988StandardforaSoftwareQualityMetricsMethodology.
IEEEStd610.12–1990StandardGlossaryofSoftwareEngineeringTerminology(ANSI.
ISO/IEC9126:1991(EInternationalStandard–InformationTechnology–Softwareproduct
evaluation–Qualitycharacteristicsandguidelinesfortheiruse.ISO/IEC8402:1986InternationalStandard–Quality–vocabulary.
ISO9000:1987InternationalStandard–Qualitymanagementandqualityassurancestandards–
Guidelinesforselectionanduse.
ISO9000–3:1991InternationalStandard–Qualitymanagementandqualityassurancestandards
–Part3:GuidelinesfortheapplicationofISO9001tothedevelopment,supplyandmaintenanceofsoftware.
ISO9001:1987InternationalStandard–Qualitysystems–Modelforqualityassurancein
design/development,production,installationandservicing.
ISO9003:1987InternationalStandard–Qualitysystems–Modelforqualityassuranceinfinal
inspectionandtest.
ISO9004:1987InternationalStandard–Qualitymanagementandqualitysystemelements–
Guidelines.
Hailstone,R.(1991Qualitymanagementandsoftwareengineering.inSoftwareQualityand
Reliability,Toolsandmethods,ed.DarrelInce,Chapman&Hall,London.Johnson,J.H.(1996Privatecommunication.
Kitchenham,B.A.,andWalker,J.G.(1986Themeaningofquality.inProc.Conf.Software
Engineering86,393–406.
OxfordEnglishDictionary,SecondEdition(1993,OxfordUniversityPress,Oxford,UK.Schach,S.R.(1990SoftwareEngineering.AksenAssociatesInc.,Homewood,IL.
Stephan,F.andMcCarthy,P.J.(1958,SamplingOpinions-AnAnalysisofSurveyProcedures.
JohnWileyandSons,NewYork
Tinnirello,P.C.(1995HandbookofApplicationDevelopment,SecondEdition.Auerbach
Publications,Boston.
Toole,F.E.(1985Subjectivemeasurementsofloudspeakersoundqualityandlistener
performance.JournaloftheAudioEngineeringSociety,33(1–2,Jan.–Feb.,2–32.
Toole,F.E.(1986aLoudspeakermeasurementsandtheirrelationshiptolistenerpreferences.I.
JournaloftheAudioEngineeringSociety,34(4,April,227–35.
Toole,F.E.(1986bLoudspeakermeasurementsandtheirrelationshiptolistenerpreferences.II
JournaloftheAudioEngineeringSociety,34(5,May,323–48.
10

BIOGRAPHY
W.MorvenGentlemanisHeadoftheSoftwareEngineeringLaboratoryoftheNationalResearchCouncilofCanada.Hehaspublishedinareasasdiverseasnumericalanalysis,computeralgebra,complexitytheory,compilertechnology,operatingsystems,computerarchitecture,real-timesystems,signalprocessing,robotics,andofcoursesoftwareengineering.Hehasbeenresponsiblefordeliveringcommercialproducts,aswellasbeinginvolvedinbuildinglargesystems.BeforeNRC,hewasfor15yearsprofessorofComputerScienceandofStatisticsattheUniversityofWaterloo,andbeforethatwasatBellTelephoneLaboratoriesinMurrayHill.HehasalsospentsabbaticalsattheNationalPhysicalLaboratoryintheUK.HisPh.D.fromPrincetonUniversityisinMathematics.
11

本文来源：https://www.2haoxitong.net/k/doc/d982f3eab8f67c1cfad6b869.html

《how do we measure it.doc》
将本文的Word文档下载到电脑，方便收藏和打印
推荐度：
点击下载文档

文档为doc格式
相
关
案
例

相关推荐

推荐内容

关于举办“庆祝中国共产党九十华诞”短讯征文大赛的通知 幼儿园大班纸工教案 加强保密工作要“四个到位” 四方会谈提纲111012 哈尔滨市2013年中考模拟题化学试卷 2015年东南大学研究生公共课中特期末复习题 构建学校、家庭、社会德育大环境 对教学型高校培养创新创业人才的思考.doc 《万花筒》名师教案 2011届高考数学最后一课
网站内容来自网络，如有侵权请联系我们，立即删除！ Copyright © 范文写作网京ICP备16605803号