Добро пожаловать в форум, Guest  >>   Войти | Регистрация | Поиск | Правила | В избранное | Подписаться
Все форумы / NoSQL, Big Data Новый топик    Ответить
 Запуск job'a в gobblin'e  [new]
x17.mstu
Member

Откуда:
Сообщений: 272
+ Запускаю job в gobblin'e
Результат журналов логов vim gobblin-current.log
2019-03-06 14:01:11 UTC INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1327 - Quartz scheduler 'LocalJobScheduler' initialized from specified file: '/opt/gobblin/gobblin-dist/conf/quartz.properties'
2019-03-06 14:01:11 UTC INFO  [SchedulerService STARTING] org.quartz.impl.StdSchedulerFactory  1331 - Quartz scheduler version: 2.2.3
2019-03-06 14:01:11 UTC INFO  [SchedulerService STARTING] org.quartz.core.QuartzScheduler  575 - Scheduler LocalJobScheduler_$_NON_CLUSTERED started.
2019-03-06 14:01:12 UTC WARN  [JobScheduler STARTING] org.apache.hadoop.util.NativeCodeLoader  62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2019-03-06 14:01:13 UTC INFO  [JobScheduler STARTING] org.apache.gobblin.scheduler.JobScheduler  507 - Scheduling configured jobs
2019-03-06 14:01:14 UTC INFO  [JobScheduler STARTING] org.apache.gobblin.scheduler.JobScheduler  527 - Loaded 1 job configurations
2019-03-06 14:01:14 UTC INFO  [JobScheduler-0] org.apache.gobblin.util.ClustersNames  74 - no default cluster mapping found
2019-03-06 14:01:14 UTC WARN  [JobScheduler-0] org.apache.gobblin.runtime.AbstractJobLauncher  223 - Creating a job specific SharedResourcesBroker. Objects will only be shared at the job level.
2019-03-06 14:01:17 UTC INFO  [JobScheduler-0] org.reflections.Reflections  229 - Reflections took 2597 ms to scan 48 urls, producing 809 keys and 2849 values
2019-03-06 14:01:18 UTC ERROR [JobScheduler-0] org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner  629 - Failed to run job GobblinKafkaQuickStart
org.apache.gobblin.runtime.JobException: Failed to run job GobblinKafkaQuickStart
        at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:437)
        at org.apache.gobblin.scheduler.JobScheduler$NonScheduledJobRunner.run(JobScheduler.java:627)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Failed to create job launcher: java.lang.RuntimeException: Failed to create FsDatasetStateStore with factory
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:158)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:107)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:85)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:68)
        at org.apache.gobblin.scheduler.JobScheduler.buildJobLauncher(JobScheduler.java:442)
        at org.apache.gobblin.scheduler.JobScheduler.runJob(JobScheduler.java:435)
        ... 4 more
Caused by: java.lang.RuntimeException: Failed to create FsDatasetStateStore with factory
        at org.apache.gobblin.runtime.FsDatasetStateStoreFactory.createStateStore(FsDatasetStateStoreFactory.java:31)
        at org.apache.gobblin.runtime.JobContext.createStateStore(JobContext.java:200)
        at org.apache.gobblin.runtime.JobContext.<init>(JobContext.java:147)
        at org.apache.gobblin.runtime.AbstractJobLauncher.<init>(AbstractJobLauncher.java:183)
        at org.apache.gobblin.runtime.local.LocalJobLauncher.<init>(LocalJobLauncher.java:81)
        at org.apache.gobblin.runtime.JobLauncherFactory.newJobLauncher(JobLauncherFactory.java:144)
        ... 9 more
Caused by: java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: instance-1.europe-west1-b.c.absolute-axis-232916.internal
        at org.apache.gobblin.runtime.FsDatasetStateStore.createStateStore(FsDatasetStateStore.java:133)
        at org.apache.gobblin.runtime.FsDatasetStateStoreFactory.createStateStore(FsDatasetStateStoreFactory.java:29)
        ... 14 more
Caused by: java.io.IOException: No FileSystem for scheme: instance-1.europe-west1-b.c.absolute-axis-232916.internal
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2304)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2311)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:90)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2350)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2332)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:369)
        at org.apache.gobblin.runtime.FsDatasetStateStore.createStateStore(FsDatasetStateStore.java:107)
        ... 15 more

Файл настройки джоба.
kafka-gobblin-hdfs.pull

###################### job configuration file ######################
job.name=GobblinKafkaQuickStart
job.group=GobblinKafka
job.description=Gobblin quick start job for Kafka
job.lock.enabled=false

kafka.brokers=0.0.0.0:6667
source.class=gobblin.source.extractor.extract.kafka.KafkaSimpleSource
extract.namespace=gobblin.extract.kafka

writer.builder.class=gobblin.writer.SimpleDataWriterBuilder
writer.file.path.type=tablename
writer.destination.type=HDFS
writer.output.format=txt

data.publisher.type=gobblin.publisher.BaseDataPublisher

mr.job.max.mappers=1
metrics.reporting.file.enabled=true
metrics.log.dir=/gobblin-kafka/metrics
metrics.reporting.file.suffix=txt

bootstrap.with.offset=earliest
fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
writer.fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;
state.store.fs.uri=instance-1.europe-west1-b.c.absolute-axis-232916.internal:8020;

mr.job.root.dir=/opt/gobblin/gobblin-dist/job_work/working
state.store.dir=/opt/gobblin/gobblin-dist/job_work/state-store
task.data.root.dir=/opt/gobblin/gobblin-dist/job_work/task-data
data.publisher.final.dir=/opt/gobblin/output

Помогите понять в чем может быть ошибка ?

Сообщение было отредактировано: 6 мар 19, 18:15
6 мар 19, 17:37    [21826607]     Ответить | Цитировать Сообщить модератору
Все форумы / NoSQL, Big Data Ответить