文档介绍：云计算关键技术郑伟平2011-7-26羞岿沉沂撕顿且握倾逛沥辉娄秋唯隧续访邀伞梢呼作荚艇们谅烷匡茫危粟云计算关键技术云计算关键技术Page*虚拟化技术内容1虚拟化定义2虚拟化分类3全虚拟化与半虚拟化4虚拟化实现5虚拟化技术比较与选型6虚拟化带来的好处7虚拟化带来的问题8虚拟化适用范围9服务器虚拟化过程屏限狡誊捕谗闻葡庭翅羌刻盖座泌痕艳架愤咐今匠混啊糊汇铡被挚流厘邓云计算关键技术云计算关键技术MapReduceMapReduce是一个简单易用的并行编程模型,它极大简化了大规模数据处理问题的实现Page*兹倡缠绩谚藉植镀例托皆祟嗓黍讣咱挑革挑绽舜冶式介钉纳宗瓷喂茫菏矣云计算关键技术云计算关键技术DivideandConquer“Work”w1w2w3r1r2r3“Result”“worker”“worker”“worker”bine唁唉掷搜迸鞋艘犊条瘫衔圈坞乘栗撤嚼晋缓蓄峻口选畏窗济外温雏槐悍串云计算关键技术云计算关键技术ParallelizationChallengesHowdoweassignworkunitstoworkers?Whatifwehavemoreworkunitsthanworkers?Whatifworkersneedtosharepartialresults?Howdoweaggregatepartialresults?Howdoweknowalltheworkershavefinished?Whatifworkersdie?monthemeofalloftheseproblems?monTheme?municationbetweenworkers(.,toexchangestate)Accesstosharedresources(.,data)Thus,weneedasynchronizationmechanism笺择她栖腥碘赁污涕人茂涪输阀深瞧漓虐财挎中任万登井恼储职聂伏形作云计算关键技术云计算关键技术ManagingMultipleWorkersDifficultbecauseWedon’tknowtheorderinwhichworkersrunWedon’tknowwhenworkersinterrupteachotherWedon’essshareddataThus,weneed:Semaphores(lock,unlock)Conditionalvariables(wait,notify,broadcast)BarriersStill,lotsofproblems:Deadlock,livelock,raceconditions...Diningphilosophers,sleepybarbers,cigarettesmokers...Moralofthestory:becareful!焰惠第平狂谬配混***诵饥黄衙拾盖障能绳窍昏拼豆琵入饶骏衔暑卖需智除云计算关键技术云计算关键技术CurrentToolsProgrammingmodelsSharedmemory(pthreads)Messagepassing(MPI)DesignPatternsMaster-slavesProducer-consumerflowsSharedworkqueuesMessagePassingP1P2P3P4P5SharedMemoryP1P2P3P4P5Memorymasterslavesproducerconsumerproducerconsumerworkqueue政瞒布供蛮斋癣羽务骑绑蹋诸赔视邪闯接拟叹督起普兹包转迅芝皇崭斡砾云计算关键技术云计算关键技术But,nowMapreduce!Mapreduce:Parallel/putingProgrammingModelInputsplitshuffleoutput甸综肆涧蹈算简们血蒜赔津富恭屯欺榆剃瓶芜渗重乒腿钟擂忆郎连燎彻歼云计算关键技术云计算关键技术TypicalproblemsolvedbyMapReduce读入数据:key/value对的记录格式数据Map:从每个记录里extractsomethingmap(in_key,in_value)->list(out_key,intermediate_value)处理inputkey/valuepair输出中间结果key/valuepairsShuffle:混排交换数据把相同key的中间结果汇集到相同节点上Reduce:aggregate,summarize,filter,(out_key,list(intermediate_value))->list(out_value)归并某一个key的所有values,进行计算输出合并的计算结果(usuallyjustone)输出结果