AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |
Back to Blog
Rapidminer studio commandline3/5/2023 ![]() Spark Resource Allocation Policy:Static,Default ConfigurationĪdvanced Spark Parameters:add the following two parameters for spark: and.Spark Archive(or libs)Path: local:///opt/huawei/Bigdata/FusionInsight_Spark2x_V100R002C80SPC200/install/FusionInsight-Spark2x-2.1.0/spark/jars.Kerberos Config File: the krb5 file downloaded from managerĪt the filter in upper right corner, search split, uncheck .maxsize.KDC Address: the KDC server IP(see the nf file).Client Principal: Kerberos user Keytab File: the keytab file downloaded from manager.Additional Libraries Directory: Spark jar files downloaded from the cluster.In RapidMiner sStadio,choose Connections->Manage Radoop Connections in the top menu.Ĭhoose New Connections->Import Hadoop Configuration Files,choose the configuration files downloaded from the cluster,click Import ConfigurationĪfter the import, click Next, go to the Connection settings window,configure ad following:.* In the FusionInsight Manager GUI, chooseService->Hive->Service Configuration`add the following configurationĭROP FUNCTION IF EXISTS r3_correlation_matrix ĭROP FUNCTION IF EXISTS r3_gaussian_rand ĭROP FUNCTION IF EXISTS r3_pivot_collect_avg ĭROP FUNCTION IF EXISTS r3_pivot_collect_count ĭROP FUNCTION IF EXISTS r3_pivot_collect_max ĭROP FUNCTION IF EXISTS r3_pivot_collect_min ĭROP FUNCTION IF EXISTS r3_pivot_collect_sum ĭROP FUNCTION IF EXISTS r3_pivot_createtable ĭROP FUNCTION IF EXISTS r3_score_naive_bayes ĬREATE FUNCTION r3_add_file AS 'eu.GenericUDFAddFile' ĬREATE FUNCTION r3_apply_model AS 'eu.GenericUDTFApplyModel' ĬREATE FUNCTION r3_correlation_matrix AS 'eu.GenericUDAFCorrelationMatrix' ĬREATE FUNCTION r3_esc AS 'eu.GenericUDFEscapeChars' ĬREATE FUNCTION r3_gaussian_rand AS 'eu.GenericUDFGaussianRandom' ĬREATE FUNCTION r3_greatest AS 'eu.GenericUDFGreatest' ĬREATE FUNCTION r3_is_eq AS 'eu.GenericUDFIsEqual' ĬREATE FUNCTION r3_least AS 'eu.GenericUDFLeast' ĬREATE FUNCTION r3_max_index AS 'eu.GenericUDFMaxIndex' ĬREATE FUNCTION r3_nth AS 'eu.GenericUDFNth' ĬREATE FUNCTION r3_pivot_collect_avg AS 'eu.GenericUDAFPivotAvg' ĬREATE FUNCTION r3_pivot_collect_count AS 'eu.GenericUDAFPivotCount' ĬREATE FUNCTION r3_pivot_collect_max AS 'eu.GenericUDAFPivotMax' ĬREATE FUNCTION r3_pivot_collect_min AS 'eu.GenericUDAFPivotMin' ĬREATE FUNCTION r3_pivot_collect_sum AS 'eu.GenericUDAFPivotSum' ĬREATE FUNCTION r3_pivot_createtable AS 'eu.GenericUDTFCreatePivotTable' ĬREATE FUNCTION r3_score_naive_bayes AS 'eu.GenericUDFScoreNaiveBayes' ĬREATE FUNCTION r3_sum_collect AS 'eu.GenericUDAFSumCollect' ĬREATE FUNCTION r3_which AS 'eu.GenericUDFWhich' ĬREATE FUNCTION r3_sleep AS 'eu.GenericUDFSleep' In the HiveServer node of the cluster,uplaod the Radoop jar files to the following path and change their owner and execution authorityĬd /opt/huawei/Bigdata/FusionInsight_HD_V100R002C80SPC200/install/FusionInsight-Hive-1.3.0/hive-1.3.0/libĬhown omm:wheel rapidminer_libs-8.2.0.jarĬd /opt/huawei/Bigdata/FusionInsight_HD_V100R002C80SPC200/install/FusionInsight-Hadoop-2.7.2/hadoop/share/hadoop/mapreduce/libĬhown omm:ficommon rapidminer_libs-8.2.0.jar Upload the jar files to each node of the cluster,eg, /usr/local/lib/radoop/ Download Radoop jars in this address the correct version。.Open yarn-site.xml,delete the following property:.Unzip the file,find the following files,copy them into a directory,like.In the Manager GUI,choose Service->Download Client->Only Configuration File.Prepare the FusionInsight client configuration filesystem and jar files For example, create a user named developuser, download the user.keytab and nf of the user and save them in your computer. Configure the local host file,file path is C:\Windows\System32\drivers\etc,add the cluster node ip and host name and save the file.Ĭreated a user with "Human-Machine" as its type( For detail, take product documentation as a reference ), grant the Hive, Spark,HDFS privileges to this user.Start rapidminer, on the top of the main menu, choose Extensions->Marketplace,type radoop,install it and restart rapidminer.Download and install RapidMiner Studio, download site.Rapidminer Studio 8.2.001 ↔ FusionInsight HD V100R002C80SPC200 (HDFS/Hive/MapReduce/Spark) preparation ¶ Connection between RapidMiner with FusionInsightHD ¶ Succeeded Case ¶
0 Comments
Read More
Leave a Reply. |