Skip to content

SQL Commiter with SCRAM-based authentication fails #64

Description

@schudoku

I am using norconex filesystem crawler 2.9.1 and want to commit to a postgre database 13.4. The database requires user authentication which is SCRAM-based. In the collector config I specified the following commiter:

  <committer class="com.norconex.committer.sql.SQLCommitter">
    <driverPath>C:\dev\databases\drivers\postgresql-42.3.5.jar</driverPath>
    <driverClass>org.postgresql.Driver</driverClass>
    <connectionUrl>jdbc:postgresql://localhost:64000/databasename</connectionUrl>
    <tableName>crawler.crawl_test</tableName>
    
    <createTableSQL>
  	CREATE TABLE {tableName} (
  		{primaryKey} VARCHAR(32672) NOT NULL,
  		content CLOB,
  		PRIMARY KEY ( {primaryKey} ),
  		title   VARCHAR(256)
  		author  VARCHAR(256)
  	)
    </createTableSQL>
    
    <createFieldSQL>
  	ALTER TABLE {tableName} ADD {fieldName} VARCHAR(5000)
    </createFieldSQL>
    
    <fixFieldValues>true</fixFieldValues>
    
    <credentials>
  	<username> ---PLAIN-USERNAME-HERE--- </username>
  	<password> ---PLAIN-PW-HERE--- </password-->
    </credentials>
    
  </committer>

And this is the stacktrace I get:

[..]
INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler finishing: committing documents.
INFO [AbstractFileQueueCommitter] Committing 266 files
INFO [SQLCommitter] Sending 100 commit operations to SQL database.
INFO [SQLCommitter] Checking if table "crawler.crawl_1" exists...
INFO [SQLCommitter] Table "crawler.crawl_1" does not exist. Attempting to create it...
INFO [AbstractCrawler] Crawler For ProSi Reports: Crawler executed in 4 seconds.
ERROR [JobSuite] Execution failed for job: Crawler For ProSi Reports
com.norconex.committer.core.CommitterException: Could not commit batch to database.
at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:480)
at com.norconex.committer.core.AbstractBatchCommitter.commitAndCleanBatch(AbstractBatchCommitter.java:179)
at com.norconex.committer.core.AbstractBatchCommitter.cacheOperationAndCommitIfReady(AbstractBatchCommitter.java:208)
at com.norconex.committer.core.AbstractBatchCommitter.commitAddition(AbstractBatchCommitter.java:143)
at com.norconex.committer.core.AbstractFileQueueCommitter.commit(AbstractFileQueueCommitter.java:222)
at com.norconex.committer.sql.SQLCommitter.commit(SQLCommitter.java:425)
at com.norconex.collector.core.crawler.AbstractCrawler.execute(AbstractCrawler.java:274)
at com.norconex.collector.core.crawler.AbstractCrawler.doExecute(AbstractCrawler.java:228)
at com.norconex.collector.core.crawler.AbstractCrawler.startExecution(AbstractCrawler.java:184)
at com.norconex.jef4.job.AbstractResumableJob.execute(AbstractResumableJob.java:49)
at com.norconex.jef4.suite.JobSuite.runJob(JobSuite.java:357)
at com.norconex.jef4.suite.JobSuite.doExecute(JobSuite.java:294)
at com.norconex.jef4.suite.JobSuite.execute(JobSuite.java:167)
at com.norconex.collector.core.AbstractCollector.start(AbstractCollector.java:150)
at com.norconex.collector.core.AbstractCollectorLauncher.launch(AbstractCollectorLauncher.java:95)
at com.norconex.collector.fs.FilesystemCollector.main(FilesystemCollector.java:76)
Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (The server requested SCRAM-based authentication, but no password was provided.)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2294)
at org.apache.commons.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:2039)
at org.apache.commons.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:1533)
at org.apache.commons.dbutils.AbstractQueryRunner.prepareConnection(AbstractQueryRunner.java:204)
at org.apache.commons.dbutils.QueryRunner.update(QueryRunner.java:419)
at com.norconex.committer.sql.SQLCommitter.ensureTable(SQLCommitter.java:607)
at com.norconex.committer.sql.SQLCommitter.commitBatch(SQLCommitter.java:463)
... 15 more
Caused by: org.postgresql.util.PSQLException: The server requested SCRAM-based authentication, but no password was provided.
at org.postgresql.core.v3.ConnectionFactoryImpl.lambda$doAuthentication$4(ConnectionFactoryImpl.java:791)
at org.postgresql.core.v3.AuthenticationPluginManager.withPassword(AuthenticationPluginManager.java:81)
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:788)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
at org.postgresql.jdbc.PgConnection.(PgConnection.java:223)
at org.postgresql.Driver.makeConnection(Driver.java:402)
at org.postgresql.Driver.connect(Driver.java:261)
at org.apache.commons.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:39)
at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:256)
at org.apache.commons.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:2304)
at org.apache.commons.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:2290)
... 21 more
INFO [JobSuite] Running Crawler For ProSi Reports: END (Mon May 23 09:41:40 CEST 2022)

It says no password provided, but I did. Could you help me?

Regards!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions