Hive nginx 또는 apache 로그 가 져 오기

1. 생 성 표
CREATE TABLE nginxlog (
 ipaddress STRING,
 identity STRING,
 user STRING,
 time STRING,
 request STRING,
 protocol STRING,
 status STRING,
 size STRING,
 referer STRING,
 agent STRING) 
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' 
WITH SERDEPROPERTIES (
 "input.regex" = "([^ ]*) ([^ ]*) ([^ ]*) (-|\\[[^\\]]*\\]) ([^ \"]*|\"[^\"]*) ([^ ]*\") (-|[0-9]*) (-|[0-9]*)(?: ([^ \"]*|\".*\") ([^ \"]*|\".*\"))?","output.format.string" = "%1$s %2$s %3$s %4$s %5$s %6$s %7$s %8$s %9$s %10$s")
STORED AS TEXTFILE; 

2. 로그 데이터 불 러 오기
load data local inpath '/home/hadoop/anjl/access.log' overwrite into table nginxlog

3. 조회
select * from nginxlog;

로그 예:
125.75.93.233 - - [14/Apr/2014:17:18:21 +0800] "GET /test/123.htm?a=1 HTTP/1.1" 200 14498 "-" -"Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
218.201.111.114 - - [14/Apr/2014:17:18:21 +0800] "GET /test/123.htm?a=1 HTTP/1.1" 200 17593 "-" -"Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"
119.180.35.121 - - [14/Apr/2014:17:18:22 +0800] "GET /test/123.htm?a=1 HTTP/1.1" 200 19492 "-" -"Apache-HttpClient/UNAVAILABLE (java 1.4)" "-"

좋은 웹페이지 즐겨찾기