利用Spark与Scala分析日志：实战教程与网站统计

日志分析

网站统计

需积分: 13 17 浏览量更新于2024-07-19 收藏 2.96MB PDF 举报

身份认证购VIP最低享 7 折!

30元优惠券

资源详情

资源推荐

www.aboutyun.com

val textFile=sc.textFile("file:///data/spark/README.md")

说明：

记得这里如果自己创建的文件可能会读取不到。报错如下

[Bash shell] 纯文本查看复制代码

001

002

003

004

005

006

007

008

009

010

011

012

013

014

015

016

017

018

019

020

021

022

023

024

025

026

027

028

029

030

031

032

033

034

035

036

java.io.FileNotFoundException: File file:/data/spark/change.txt does not exist

at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)

at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)

at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)

at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)

at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)

at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)

at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108)

at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)

at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:240)

at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:211)

at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)

at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)

at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)

at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)

at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)

at org.apache.spark.scheduler.Task.run(Task.scala:89)

at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:

org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1431)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1419)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1418)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1418)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:799)

at scala.Option.foreach(Option.scala:236)

www.aboutyun.com

037

038

039

040

041

042

043

044

045

046

047

048

049

050

051

052

053

054

055

056

057

058

059

060

061

062

063

064

065

066

067

068

069

070

071

072

073

074

075

076

077

078

079

080

at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:799)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1640)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1599)

at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1588)

at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)

at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:620)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1832)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1845)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1858)

at org.apache.spark.SparkContext.runJob(SparkContext.scala:1929)

at org.apache.spark.rdd.RDD.count(RDD.scala:1157)

at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30)

at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35)

at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37)

at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:39)

at $iwC$$iwC$$iwC$$iwC.<init>(<console>:41)

at $iwC$$iwC$$iwC.<init>(<console>:43)

at $iwC$$iwC.<init>(<console>:45)

at $iwC.<init>(<console>:47)

at <init>(<console>:49)

at .<init>(<console>:53)

at .<clinit>(<console>)

at .<init>(<console>:7)

at .<clinit>(<console>)

at $print(<console>)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:497)

at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)

at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346)

at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)

at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)

at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)

at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)

at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)

at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)

at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)

at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)

at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)

org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)

at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)

剩余36页未读，继续阅读

princemortor

粉丝: 13
资源: 266

利用Spark与Scala分析日志：实战教程与网站统计

图解网站分析-pdf

网站分析实战-王彦平

【图像融合】基于matlab深度学习医学图像融合【含Matlab源码 8038期】.md

【图像配准】基于matlab SIFT图像配准【含Matlab源码 1007期】.md

基于 YOLOv5（假设为 YOLOv11）的火灾火焰烟雾检测系统（包含详细的完整的程序和数据）

ahds-0.2.3-cp39-cp39-win_amd64.whl

【图像加密】基于matlab预测误差分类置乱图像加密解密【含Matlab源码 1846期】.md

【图像评价】基于matlab无参考图像质量评价NIQE【含Matlab源码 681期】.md

bsdiff4-1.1.9-cp35-cp35m-win_amd64.whl

pycosat-0.6.3-cp39-cp39-win_amd64.whl

wrapt-1.11.2-cp35-cp35m-win_amd64.whl

【图像修复】基于matlab小波变换DWT图像修复（含MSE PSNR）【含Matlab源码 3503期】.md

crc16-0.1.1-cp310-cp310-win_amd64.whl

VBA利用CopyMemory函数复制数组【二维数组转一维数组】.xlsm

fastcache-1.1.0-cp38-cp38-win_amd64.whl

Vue.js 是一个流行的前端 JavaScript 框架，用于构建用户界面 它因其简洁的 API 和易于学习的特性而受到广泛欢

yappi-1.3.5-cp39-cp39-win_amd64.whl

【图像加密解密】线性正则变换与菲涅尔变换图像加密解密（含MSE PSNR）【含Matlab源码 4704期】.md

HengCe-18900-2024-2030全球与中国先进封装市场现状及未来发展趋势 Sample-样本V2(2).docx

glcontext-2.3.5-cp38-cp38-win_amd64.whl

最新资源

Vue.js 是一个流行的前端 JavaScript 框架，用于构建用户界面它因其简洁的 API 和易于学习的特性而受到广泛欢