hive入门零基础教程
手把手教你在腾讯云上搭建hive3.1.2的方法环境准备
所有操作开始前，先确定hadoop版本已经装好。具体可以看我的另一篇博客
搭建hadoop3.x
mysql安装
先把安装包放到software文件夹，并解压到mysql-lib中
|
|
[root@master software]# ll总用量 1422372-rw-r--r-- 1 root root 312850286 7月 2 15:00 apache-hive-3.1.2-bin.tar.gz-rw-r--r-- 1 root root 338075860 7月 2 09:39 hadoop-3.1.3.tar.gz-rw-r--r-- 1 root root 195013152 7月 2 09:36 jdk-8u212-linux-x64.tar.gz-rw-r--r-- 1 root root 609556480 7月 2 15:08 mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar-rw-r--r-- 1 root root 985600 7月 2 14:44 mysql-connector-java-5.1.37.jar[root@master software]# mkdir mysql-lib[root@master software]# tar -xf mysql-5.7.28-1.el7.x86_64.rpm-bundle.tar -C ./mysql-lib/[root@master software]# cd mysql-lib/[root@master mysql-lib]# ll总用量 595272-rw-r--r-- 1 7155 31415 45109364 9月 30 2019 mysql-community-client-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 318768 9月 30 2019 mysql-community-common-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 7037096 9月 30 2019 mysql-community-devel-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 49329100 9月 30 2019 mysql-community-embedded-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 23354908 9月 30 2019 mysql-community-embedded-compat-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 136837816 9月 30 2019 mysql-community-embedded-devel-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 4374364 9月 30 2019 mysql-community-libs-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 1353312 9月 30 2019 mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 208694824 9月 30 2019 mysql-community-server-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 133129992 9月 30 2019 mysql-community-test-5.7.28-1.el7.x86_64.rpm |
先处理一下环境，因为centos自带mariadb，会产生冲突，所以要先卸载
|
|
[root@master mysql-lib]# rpm -qa|grep mariadbmariadb-libs-5.5.56-2.el7.x86_64[root@master mysql-lib]# sudo rpm -e --nodeps mariadb-libs # 查看一下是不是卸载干净了[root@master mysql-lib]# rpm -qa|grep mariadb[root@master mysql-lib]# rpm -qa|grep mysql |
没什么问题了就可以安装了，一定要按顺序安装
|
|
[root@master mysql-lib]# ll总用量 595284-rw-r--r-- 1 7155 31415 45109364 9月 30 2019 mysql-community-client-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 318768 9月 30 2019 mysql-community-common-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 7037096 9月 30 2019 mysql-community-devel-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 49329100 9月 30 2019 mysql-community-embedded-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 23354908 9月 30 2019 mysql-community-embedded-compat-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 136837816 9月 30 2019 mysql-community-embedded-devel-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 4374364 9月 30 2019 mysql-community-libs-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 1353312 9月 30 2019 mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 208694824 9月 30 2019 mysql-community-server-5.7.28-1.el7.x86_64.rpm-rw-r--r-- 1 7155 31415 133129992 9月 30 2019 mysql-community-test-5.7.28-1.el7.x86_64.rpm[root@master mysql-lib]# sudo rpm -ivh mysql-community-common-5.7.28-1.el7.x86_64.rpm警告：mysql-community-common-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY准备中... ################################# [100%]正在升级/安装... 1:mysql-community-common-5.7.28-1.e################################# [100%][root@master mysql-lib]# sudo rpm -ivh mysql-community-libs-5.7.28-1.el7.x86_64.rpm警告：mysql-community-libs-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY准备中... ################################# [100%]正在升级/安装... 1:mysql-community-libs-5.7.28-1.el7################################# [100%][root@master mysql-lib]# sudo rpm -ivh mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm警告：mysql-community-libs-compat-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY准备中... ################################# [100%]正在升级/安装... 1:mysql-community-libs-compat-5.7.2################################# [100%][root@master mysql-lib]# sudo rpm -ivh mysql-community-client-5.7.28-1.el7.x86_64.rpm警告：mysql-community-client-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY准备中... ################################# [100%]正在升级/安装... 1:mysql-community-client-5.7.28-1.e################################# [100%] |
安装到第5个时，如果centos是最小化安装的，那会报这个问题
|
|
[root@master mysql-lib]# sudo rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm警告：mysql-community-server-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY错误：依赖检测失败： libnuma.so.1()(64bit) 被 mysql-community-server-5.7.28-1.el7.x86_64 需要 libnuma.so.1(libnuma_1.1)(64bit) 被 mysql-community-server-5.7.28-1.el7.x86_64 需要 libnuma.so.1(libnuma_1.2)(64bit) 被 mysql-community-server-5.7.28-1.el7.x86_64 需要 |
|
|
[root@master mysql-lib]# yum install -y libaio[root@master mysql-lib]# yum install -y numactl |
然后再尝试安装就没问题了
|
|
[root@master mysql-lib]# sudo rpm -ivh mysql-community-server-5.7.28-1.el7.x86_64.rpm警告：mysql-community-server-5.7.28-1.el7.x86_64.rpm: 头V3 DSA/SHA1 Signature, 密钥 ID 5072e1f5: NOKEY准备中... ################################# [100%]正在升级/安装... 1:mysql-community-server-5.7.28-1.e################################# [100%] |
清理历史残余文件
|
|
[root@master mysql-lib]# cat /etc/my.cnf# For advice on how to change settings please see# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html[mysqld]## Remove leading # and set to the amount of RAM for the most important data# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.# innodb_buffer_pool_size = 128M## Remove leading # to turn on a very important data integrity option: logging# changes to the binary log between backups.# log_bin## Remove leading # to set options mainly useful for reporting servers.# The server defaults are faster for transactions and fast SELECTs.# Adjust sizes as needed, experiment to find the optimal values.# join_buffer_size = 128M# sort_buffer_size = 2M# read_rnd_buffer_size = 2Mdatadir=/var/lib/mysqlsocket=/var/lib/mysql/mysql.sock# Disabling symbolic-links is recommended to prevent assorted security riskssymbolic-links=0log-error=/var/log/mysqld.logpid-file=/var/run/mysqld/mysqld.pid |
看到了mysql目录是/var/lib/mysql
|
|
#进入这个地址 看看是否有历史残余文件[root@master mysql-lib]# cd /var/lib/mysql# 没东西就没问题，有就都删了[root@master mysql]# ll总用量 0 |
初始化mysql
|
|
[root@master mysql]# sudo mysqld --initialize --user=mysql |
看下默认密码
|
|
[root@master mysql]# cat /var/log/mysqld.log 2020-07-02T07:29:10.095244Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).2020-07-02T07:29:11.331738Z 0 [Warning] InnoDB: New log files created, LSN=457902020-07-02T07:29:11.498492Z 0 [Warning] InnoDB: Creating foreign key constraint system tables.2020-07-02T07:29:11.590436Z 0 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: b94654a6-bc35-11ea-a160-525400f3e388.2020-07-02T07:29:11.600554Z 0 [Warning] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.2020-07-02T07:29:13.093996Z 0 [Warning] CA certificate ca.pem is self signed.2020-07-02T07:29:13.522160Z 1 [Note] A temporary password is generated for root@localhost: p;oK<M)<_8p/ |
密码是p;oK<M)<_8p/，复制一下，进入mysql
进入mysql
启动mysql服务
|
|
[root@master mysql]# systemctl start mysqld[root@master mysql]# systemctl status mysqld● mysqld.service - MySQL Server Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled) Active: active (running) since 四 2020-07-02 15:31:00 CST; 10s ago Docs: man:mysqld(8) http://dev.mysql.com/doc/refman/en/using-systemd.html Process: 5617 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS) Process: 5497 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS) Main PID: 5624 (mysqld) Tasks: 27 Memory: 205.4M CGroup: /system.slice/mysqld.service └─5624 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.p...7月 02 15:30:55 master systemd[1]: Starting MySQL Server...7月 02 15:31:00 master systemd[1]: Started MySQL Server. |
进入mysql
|
|
$ mysql -u root -p |
提示输入密码，输入刚刚复制的密码
成功进入mysql后，设置我们的mysql密码，这个自己记住哈~
|
|
#设置密码为你的密码mysql> set password = password("你的密码"); |
接下来设置root用户的权限，允许任意ip连接，为后面的hive连接做准备
|
|
mysql> use mysql;mysql> select host,user from user;+-----------+---------------+| host | user |+-----------+---------------+| localhost | mysql.session || localhost | mysql.sys || localhost | root |+-----------+---------------+3 rows in set (0.00 sec)mysql> update mysql.user set host='%' where user='root';Query OK, 1 row affected (0.00 sec)Rows matched: 1 Changed: 1 Warnings: 0# 权限高的在最上面（新版本优化的）mysql> select host,user from user;+-----------+---------------+| host | user |+-----------+---------------+| % | root || localhost | mysql.session || localhost | mysql.sys |+-----------+---------------+3 rows in set (0.00 sec)--这步一定要做，是刷新配置的，不然密码无法生效mysql> flush privileges;Query OK, 0 rows affected (0.00 sec)mysql> quit |
然后再尝试用更改后的密码登录一下，成功了就是配置好了。
安装Hive
到/home/centos/software目录下，解压apache-hive-3.1.2-bin.tar.gz到/home/centos/module/目录下面
|
|
[root@master software]# tar -zxvf apache-hive-3.1.2-bin.tar.gz -C /home/centos/module/ |
进入module文件夹，把文件名改一下
|
|
[root@master module]# ll总用量 12drwxr-xr-x 9 root root 4096 7月 2 15:40 apache-hive-3.1.2-bindrwxr-xr-x 11 centos centos 4096 7月 2 11:51 hadoop-3.1.3drwxr-xr-x 7 10 143 4096 4月 2 2019 jdk1.8.0_212[root@master module]# mv apache-hive-3.1.2-bin/ hive[root@master module]# ll总用量 12drwxr-xr-x 11 centos centos 4096 7月 2 11:51 hadoop-3.1.3drwxr-xr-x 9 root root 4096 7月 2 15:40 hivedrwxr-xr-x 7 10 143 4096 4月 2 2019 jdk1.8.0_212 |
配置环境变量
|
|
[root@master module]# vim /etc/profile.d/my_env.sh |
|
|
#HIVE_HOMEexport HIVE_HOME=/home/centos/module/hiveexport PATH=$PATH:$HIVE_HOME/bin |
加载环境变量，并验证是否成功
|
|
[root@master module]# source /etc/profile[root@master module]# echo $HIVE_HOME/home/centos/module/hive |
解决日志Jar包冲突
hadoop和hive之间的log4j版本冲突，要把hive的lib文件夹中的jar包名字改一下，无法启用即可。就直接在文件名后加.bak即可。
|
|
[atguigu@hadoop102 lib]$ mv log4j-slf4j-impl-2.10.0.jar log4j-slf4j-impl-2.10.0.jar.bak[atguigu@hadoop102 lib]$ ll | grep log4j-rw-rw-r--. 1 atguigu atguigu 63835 4月 15 00:34 log4j-1.2-api-2.10.0.jar-rw-rw-r--. 1 atguigu atguigu 255485 4月 15 00:24 log4j-api-2.10.0.jar-rw-rw-r--. 1 atguigu atguigu 1597622 4月 15 00:24 log4j-core-2.10.0.jar-rw-rw-r--. 1 atguigu atguigu 24173 4月 15 00:24 log4j-slf4j-impl-2.10.0.jar.bak-rw-rw-r--. 1 atguigu atguigu 32060 4月 15 00:34 log4j-web-2.10.0.jar |
hive元数据配置到mysql
将MySQL的JDBC驱动拷贝到Hive的lib目录下
|
|
[root@master lib]# cp /home/centos/software/mysql-connector-java-5.1.37.jar ./ |
看看是否拷贝成功
|
|
[root@master lib]# ll | grep mysql-con-rw-r--r-- 1 root root 985600 7月 2 15:47 mysql-connector-java-5.1.37.jar |
配置metastore到mysql
在$HIVE_HOME/conf目录下新建hive-site.xml文件，
|
|
[root@master conf]# vim hive-site.xml |
有三个值需要你改一下
- 腾讯云内网ip
- mysql密码
- 本机hostname