Programming Pig: The Ultimate Guide for Learning Pig Latin

需积分: 9 123 浏览量更新于2024-07-24 收藏 6.41MB PDF 举报

"Programming Pig"是一本由Alan Gates编写的关于Pig编程的英文版书籍，由O'Reilly Media出版。这本书全面介绍了Pig语言和工具，适用于学习和使用Pig的人员，特别是对于那些希望避免翻译错误的读者来说，原版书籍提供了最准确的信息。 Pig是Apache Hadoop项目的一部分，它提供了一个高级平台，用于处理和分析大规模数据集。Pig Latin是Pig的语言，设计得简单易学，允许用户通过一系列操作（称为“猪存根”或“猪逻辑”）来处理数据，而无需深入理解底层MapReduce机制。这本书详细讲解了如何使用Pig Latin编写数据处理脚本，以及Pig在大数据分析中的应用。书中可能涵盖了以下几个主要知识点： 1. **Pig Latin基础**：包括Pig Latin的基本语法、数据类型、加载和存储数据的操作、数据转换函数等。这些基础知识是理解和编写Pig脚本的基础。 2. **数据处理流程**：详细解释了如何使用Pig进行数据清洗、转换和聚合，以及如何通过JOIN操作处理多个数据集。 3. **Pig与Hadoop的集成**：阐述Pig如何与Hadoop生态系统其他组件（如HDFS和HBase）协同工作，以及如何在分布式环境中运行Pig脚本。 4. **性能优化**：介绍如何通过参数调整、优化逻辑和并行化策略提高Pig作业的执行效率。 5. **高级特性**：可能包括自定义函数（UDFs）、 PigStreaming、 Pig with Tez 或 YARN 等，让读者了解如何扩展Pig的功能。 6. **案例研究**：通过实际案例展示Pig在大数据分析项目中的应用，帮助读者将理论知识应用于实践。 7. **错误处理和调试**：教导读者如何识别和解决Pig脚本中的错误，以及如何调试Pig作业。 8. **最佳实践**：分享编写高效、可维护的Pig脚本的技巧和建议。此外，书中还可能包含一个索引和在线更新信息，以供读者查询特定主题或获取最新的修正信息。对于那些在企业或教育环境中使用Pig的人来说，这是一本宝贵的参考书，可以帮助他们提升大数据处理的能力。

Douglas of the Hadoop project provided me with very helpful feedback on the sections

covering Hadoop and MapReduce.

I would also like to thank Mike Loukides and the entire team at O’Reilly. They have

made writing my first book an enjoyable and exhilarating experience. Finally, thanks

to Yahoo! for nurturing Pig and dedicating more than 25 engineering years (and still

counting) of effort to it, and for graciously giving me the time to write this book.

xiv | Preface

Part of the specification of a MapReduce job is the key on which data will be collected.

For example, if you were processing web server logs for a website that required users

to log in, you might choose the user ID to be your key so that you could see everything

done by each user on your website. In the shuffle phase, which happens after the map

phase, data is collected together by the key the user has chosen and distributed to

different machines for the reduce phase. Every record for a given key will go to the same

reducer.

In the reduce phase, the application is presented each key, together with all of the

records containing that key. Again this is done in parallel on many machines. After

processing each group, the reducer can write its output. See the next section for a

walkthrough of a simple MapReduce program. For more details on how MapReduce

works, see “MapReduce” on page 189.

MapReduce’s hello world

Consider a simple MapReduce application that counts the number of times each word

appears in a given text. This is the “hello world” program of MapReduce. In this ex-

ample the map phase will read each line in the text, one at a time. It will then split out

each word into a separate string, and, for each word, it will output the word and a 1 to

indicate it has seen the word one time. The shuffle phase will use the word as the key,

hashing the records to reducers. The reduce phase will then sum up the number of

times each word was seen and write that together with the word as output. Let’s con-

sider the case of the nursery rhyme “Mary Had a Little Lamb.” Our input will be:

Mary had a little lamb

its fleece was white as snow

and everywhere that Mary went

the lamb was sure to go.

Let’s assume that each line is sent to a different map task. In reality, each map is assigned

much more data than this, but it makes the example easier to follow. The data flow

through MapReduce is shown in Figure 1-1.

Once the map phase is complete, the shuffle phase will collect all records with the same

word onto the same reducer. For this example we assume that there are two reducers:

all words that start with A-L are sent to the first reducer, and M-Z are sent to the second

reducer. The reducers will then output the summed counts for each word.

Pig uses MapReduce to execute all of its data processing. It compiles the Pig Latin scripts

that users write into a series of one or more MapReduce jobs that it then executes. See

Example 1-1 for a Pig Latin script that will do a word count of “Mary Had a Little

Lamb.”

2 | Chapter 1: Introduction

Pig Latin, a Parallel Dataflow Language

Pig Latin is a dataflow language. This means it allows users to describe how data from

one or more inputs should be read, processed, and then stored to one or more outputs

in parallel. These data flows can be simple linear flows like the word count example

given previously. They can also be complex workflows that include points where mul-

tiple inputs are joined, and where data is split into multiple streams to be processed by

different operators. To be mathematically precise, a Pig Latin script describes a directed

acyclic graph (DAG), where the edges are data flows and the nodes are operators that

process the data.

This means that Pig Latin looks different from many of the programming languages

you have seen. There are no if statements or for loops in Pig Latin. This is because

traditional procedural and object-oriented programming languages describe control

flow, and data flow is a side effect of the program. Pig Latin instead focuses on data

flow. For information on how to integrate the data flow described by a Pig Latin script

with control flow, see Chapter 9.

Comparing query and dataflow languages

After a cursory look, people often say that Pig Latin is a procedural version of SQL.

Although there are certainly similarities, there are more differences. SQL is a query

language. Its focus is to allow users to form queries. It allows users to describe what

question they want answered, but not how they want it answered. In Pig Latin, on the

other hand, the user describes exactly how to process the input data.

Another major difference is that SQL is oriented around answering one question. When

users want to do several data operations together, they must either write separate quer-

ies, storing the intermediate data into temporary tables, or write it in one query using

subqueries inside that query to do the earlier steps of the processing. However, many

SQL users find subqueries confusing and difficult to form properly. Also, using sub-

queries creates an inside-out design where the first step in the data pipeline is the in-

nermost query.

Pig, however, is designed with a long series of data operations in mind, so there is no

need to write the data pipeline in an inverted set of subqueries or to worry about storing

data in temporary tables. This is illustrated in Examples 1-2 and 1-3.

Consider a case where a user wants to group one table on a key and then join it with a

second table. Because joins happen before grouping in a SQL query, this must be ex-

pressed either as a subquery or as two queries with the results stored in a temporary

table. Example 1-3 will use a temporary table, as that is more readable.

4 | Chapter 1: Introduction

剩余221页未读，继续阅读

hoh2009

粉丝: 0
资源: 2

Programming Pig: The Ultimate Guide for Learning Pig Latin

Pig编程指南

Pig编程指南中文版

《Pig编程指南》一1.1 Pig是什么？

pig 的基本框架与实现

用Python定义基类animal，子类dog和子类pig，编程实现调用类dog的run和类pig的run的不同，测试类的多态性

定义基类animal，子类dog和子类pig，编程实现 调用类dog的run和类pig的run的不同，测试类的 多态性

介绍W25Q32FV2PIG的芯片参数和功能特点

Pig是一个高可靠易扩展可伸缩的列式数据库

mapreduceMapReduce编程初级实践

最新资源

《Pig编程指南》一1.1　Pig是什么？

定义基类animal，子类dog和子类pig，编程实现调用类dog的run和类pig的run的不同，测试类的多态性