There is a new KNIME forum. You can still browse and read content from our old forum but if you want to create new posts or join ongoing discussions, please visit our new KNIME forum: https://forum.knime.com

Knime 3.5.x S3 file picker issue

Member for

1 year 10 months mweber

Since Knime 3.5.x we face an issue with the S3 file picker when there are deleted prefixes in S3.

The error can be reproduced the following way: Upload file to S3, then delete the prefix.

aws s3 cp LocalFile s3://bucketname/aaa/RemoteFile
aws s3 rm s3://bucketname/aaa/
aws s3 ls s3://bucketname/aaa/

Note that the rm deletes only the prefix "aaa/", not the complete structure. (S3 is a object storage, that is, there are no directories as in Windows/Linux filesystems).

Now when it comes to the S3 file picker, as soon as "bucketname" is expanded, it gets an error:

WARN  Amazon S3 File Picker 2:2        Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: D57EEF4ED3CD8528)

The S3 file picker becomes unusable. We traced it down and found out that a HEAD Object Request is sent to AWS, the AWS API responds with a HTTP 403.

The following stacktrace is shown in the knime.log

2018-03-02 08:13:13,157 : INFO : SwingWorker-pool-4-thread-10 : S3Connection : Amazon S3 File Picker : 2:2 : Create a new AmazonS3Client in Region "eu-central-1" with connection timeout 30000 milliseconds
2018-03-02 08:13:16,955 : WARN  : SwingWorker-pool-4-thread-2 : RemoteFileChooser : Amazon S3 File Picker : 2:2 : Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: D57EEF4ED3CD8528)
2018-03-02 08:13:16,956 : DEBUG : SwingWorker-pool-4-thread-2 : RemoteFileChooser : Amazon S3 File Picker : 2:2 : Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: D57EEF4ED3CD8528)
com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: D57EEF4ED3CD8528), S3 Extended Request ID: orSlGF4OByv0GvKNX4zbUeEWuxt6ENb7EAXYWR94scIRQnsUgHqESEAovHMCld2Hmwhl9004Dv4=
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1588)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1258)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1030)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:742)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:716)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
        at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
        at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4221)
        at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4168)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1249)
        at com.amazonaws.services.s3.AmazonS3Client.getObjectMetadata(AmazonS3Client.java:1224)
        at com.amazonaws.services.s3.AmazonS3Client.doesObjectExist(AmazonS3Client.java:1284)
        at org.knime.cloud.aws.s3.filehandler.S3RemoteFile.doestBlobExist(S3RemoteFile.java:139)
        at org.knime.cloud.core.file.CloudRemoteFile.exists(CloudRemoteFile.java:216)
        at org.knime.base.filehandling.remote.files.RemoteFile.getPath(RemoteFile.java:275)
        at org.knime.base.filehandling.remote.files.RemoteFile.getFullName(RemoteFile.java:252)
        at org.knime.base.filehandling.remote.dialog.RemoteFileChooser$RemoteFileTreeNode.<init>(RemoteFileChooser.java:619)
        at org.knime.base.filehandling.remote.dialog.RemoteFileChooser$RemoteFileTreeNodeWorker.doInBackgroundWithContext(RemoteFileChooser.java:495)
        at org.knime.base.filehandling.remote.dialog.RemoteFileChooser$RemoteFileTreeNodeWorker.doInBackgroundWithContext(RemoteFileChooser.java:1)
        at org.knime.core.util.SwingWorkerWithContext.doInBackground(SwingWorkerWithContext.java:106)
        at javax.swing.SwingWorker$1.call(SwingWorker.java:295)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at javax.swing.SwingWorker.run(SwingWorker.java:334)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Comments