I think this tutorial on Yahoo’s developer network will help you with this: Chaining Jobs You use the JobClient.runJob(). The output path of the data from the first job becomes the input path to your second job. These need to be passed in as arguments to your jobs with appropriate code to parse them and … Read more
Specify split column manually. Split column is not necessarily equal to PK. You can have complex PK and some int Split column. You can specify any integer column or even simple function (some simple function like substring or cast, not aggregation or analytic). Split column preferably should be evenly distributed integer. For example if your … Read more
Functionality wise there is not much difference between the old (o.a.h.mapred) and the new (o.a.h.mapreduce) API. The only significant difference is that records are pushed to the mapper/reducer in the old API. While the new API supports both pull/push mechanism. You can get more information about the pull mechanism here. Also, the old API has … Read more
You can use hdfs fsck / to determine which files are having problems. Look through the output for missing or corrupt blocks (ignore under-replicated blocks for now). This command is really verbose especially on a large HDFS filesystem so I normally get down to the meaningful output with hdfs fsck / | egrep -v ‘^\.+$’ … Read more
In order to forcefully let the namenode leave safemode, following command should be executed: bin/hadoop dfsadmin -safemode leave You are getting Unknown command error for your command as -safemode isn’t a sub-command for hadoop fs, but it is of hadoop dfsadmin. Also after the above command, I would suggest you to once run hadoop fsck … Read more
There was no solution with python code and I recently had to read zips in pyspark. And, while searching how to do that I came across this question. So, hopefully this’ll help others. import zipfile import io def zip_extract(x): in_memory_data = io.BytesIO(x[1]) file_obj = zipfile.ZipFile(in_memory_data, “r”) files = [i for i in file_obj.namelist()] return dict(zip(files, … Read more
From Apache docs The ZKFailoverController (ZKFC) is a new component which is a ZooKeeper client which also monitors and manages the state of the NameNode. Each of the machines which runs a NameNode also runs a ZKFC, and that ZKFC is responsible for: Health monitoring – the ZKFC pings its local NameNode on a periodic … Read more
Here is the approach i used to solved this problem using hive’s internal UDF function, “map”: select b.id, b.code, concat_ws(”,b.p) as p, concat_ws(”,b.q) as q, concat_ws(”,b.r) as r, concat_ws(”,b.t) as t from ( select id, code, collect_list(a.group_map[‘p’]) as p, collect_list(a.group_map[‘q’]) as q, collect_list(a.group_map[‘r’]) as r, collect_list(a.group_map[‘t’]) as t from ( select id, code, map(proc1,proc2) as … Read more
Hive has a relational database on the master node it uses to keep track of state. For instance, when you CREATE TABLE FOO(foo string) LOCATION ‘hdfs://tmp/’;, this table schema is stored in the database. If you have a partitioned table, the partitions are stored in the database(this allows hive to use lists of partitions without … Read more
The number of map tasks for a given job is driven by the number of input splits and not by the mapred.map.tasks parameter. For each input split a map task is spawned. So, over the lifetime of a mapreduce job the number of map tasks is equal to the number of input splits. mapred.map.tasks is … Read more