Questions tagged [amazon-s3]

Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services.

0
votes
0answers
7 views

Different ways of reading data from s3 bucket using boto3

I'm trying to read some json file from s3 bucket and then trying to post the data into rds and redshift(using a post api developed/intended for that). I approached to do this in 2 ways using boto3. ...
0
votes
0answers
5 views

Accessing Multiple COS instance under the same IBM account

I am trying to access multiple COS instances which are there under the same IBM account. I see that each COS instance is having different accesskey,secret-key. The property in hive.xml(fs.s3a.access....
0
votes
0answers
7 views

Encountering `Cannot find module webpack/lib/removeAndDo` when deploying Angular project onto S3 from Jenkins

I am trying to deploy my Angular project on AWS as part of one my stages in my Jenkins job. Long story short, I run some tests and then deploy the project using npm run build:serverless:deploy. When I ...
0
votes
1answer
16 views

What is the best way to convert a file in amazon s3 with Django/python?

I am making a website to get to know aws and Django better. The idea is to let a user upload an excel file, convert it to csv and then let the user download the converted csv file. I am using amazon ...
0
votes
1answer
13 views

sqlite3: Connect to a database in cloud (S3)

I have a little sqlite database(110kb) in a s3 bucket. I want to connect to that database every time I run my python application. An option is just to download database everytime I run the python ...
0
votes
1answer
12 views

Stop S3 recursive event using custom header parameters

I have an event associated with S3 PUTS that will trigger a lambda. This lambda in turn updates the same object in S3. I want to avoid recursive S3 updates. Can I pass any custom header value as part ...
0
votes
2answers
11 views

How to create a custom event trigger to invoke a lambda whenever a new bucket is created?

I have a lambda function in Python that I want to invoke whenever a new s3 bucket is created. I want to create a custom event trigger to invoke it. What would be the best way to go ahead implementing ...
0
votes
0answers
22 views

Javascript - Access-Control-Allow-Origin missing only in Firefox (using AWS S3 and Cloudfront)

Research I have a very odd issue occurring on my site. I have searched around, namely the following articles, but none of these seem to solve my issue. Unable to load cross-origin image (from ...
0
votes
0answers
6 views

JupyterHub server is unable start in Terraformed EMR cluster running in private subnet

I'm creating an EMR cluster (emr-5.24.0) with Terraform, deployed into a private subnet, that includes Spark, Hive and JupyterHub. I've added an additional configuration JSON to the deployment, which ...
1
vote
1answer
17 views

S3 notification creates multiple events

We have been using AWS S3 notifications to trigger lambda functions when files land on S3 and this model has worked reasonably well until we noticed that some files are processed multiple times, ...
0
votes
0answers
9 views

how to achieve idempotent behavior in lambda that writes to S3?

I have a AWS lambda that does some calculation and writes output to S3 location. The AWS lambda is triggered by cloudwatch cron expression. Since the lambda can be triggered multiple times, I want to ...
0
votes
2answers
21 views

AWS pricing for S3 object download using site-to-site VPN

I would like to understand the monthly cost of using a VPN site-to-site link to transport data from S3 to my on-premises host. I will use AWS Lambda to transfer around 12 TB of data each month from S3....
0
votes
0answers
8 views

Best sequence when uploading large files to different cloud services

I am trying to upload large files (100+ Gb) to several cloud storage services from a Django app. I have written a first view to do this for S3 and it is working: class FilePolicyAPI(APIView): ... I ...
0
votes
0answers
6 views

How to parse xml S3 bucket in python

Fail to parse xml from s3 bucket This is response from S3 bucket: <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Name>example-bucket</Name> <Prefix>...
0
votes
1answer
35 views

How to download file from amazon s3 using only c#(Without using AWS SDK)

My requirement is download a files from Amazon s3 with out using Aws sdk for C# and using restfull(HttpWebRequest & HttpWebResponse) api calls. I got response after successfully uploaded the ...
1
vote
0answers
12 views

Is there a way to make a private connection between a Heroku Private Space and AWS S3?

Is there a way to access S3 privately (not over the internet) from a Heroku Private Space? Using Private Space Peering and a HTTP proxy is not an option.
0
votes
1answer
12 views

Assume IAM role and then generate pre-signed URL in Python

I am trying to assume an IAM role and then use Amazon S3 by generating a presigned URL in order to access an S3 bucket in it. This is how I have configured my code in Python : def create_dynamicurl(...
0
votes
2answers
16 views

AccessDenied on AWS Athena

When I run a simple select * query on AWS Athena I get an access denied error. com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: ...
0
votes
1answer
16 views

What is the correct syntax for Cloudformation AWS::IAM::Policy for S3 full access

In a CloudFormation script I create an IAM::Policy that gets applied to a Kinesis Firehose. It has the following PolicyDocument Statement Statement: - Effect: Allow Action: - 's3:...
0
votes
0answers
12 views

How to provide bit bucket pipeline user access to S3?

I want to setup deployment using bitbucket pipeline and AWS cloud formation. which I am able to achieve but I was able to do this with public access. How can I deploy with rights only to bitbucket ...
-1
votes
0answers
10 views

Pull images from S3 [duplicate]

Trying to display .PNG file in Flask app based on filename. How can I pull a file from S3 into my flask app based upon a filename entered by the user on a web page? I am also concurrently pulling in ...
1
vote
0answers
16 views

AWS S3 Rest API: Unable to upload object

I am trying to upload an object into a bucket in s3. I am using the REST API (no SDK, as there is no SDK for Salesforce) I have follow the documentation: https://docs.aws.amazon.com/AmazonS3/latest/...
0
votes
1answer
14 views

Copy files between s3 Buckets with destination bucket having a directory

I have two s3 buckets. I want to copy a file from Bucket from to Bucket to. In Bucket to, I have a folder /copy_here. When I tried to directly copy the contents between the buckets using Boto it works....
0
votes
1answer
19 views

AWS s3 sync to upload if file does not exist in target

I have uploaded about 1,000,000 files from my local directory to s3 buckets/subfolders and some of them have failed. I would like to use the 'sync' option to capture those that did not make it the ...
1
vote
2answers
36 views

AWS Athena: cross account write of CTAS query result

I have big historical dataset in an account A. This dataset is in csv format and partitioned by year/month/day/hour/. My goal is to convert this data to parquet, with additional normalisation steps ...
0
votes
0answers
8 views

How do I add a bucket name to s3.Bucket in aws cdk

I can create an aws cdk s3.CfnBucket with a "bucket_name" argument and it looks good. Does the "s3.Bucket" cdn construct have a bucketName property? It is documented as a cdk.PhysicalName property, ...
0
votes
0answers
8 views

How to change the region of AWS at runtime using iOS sdk?

Thanks in advance. I'm planning to release an app in different countries, where the users will be given an option to select the country and based on the selection their respective aws region will be ...
0
votes
1answer
13 views

How to set S3SinkConnector (Failed to find class)

How can i set S3SinkConnector to stream data from my topic to S3 bucket? I've download plugin from confluent hub, created a file connect-standalone.properties and s3-sink.properties. When I try to ...
0
votes
1answer
11 views

save a csv file into s3 bucket from pypark dataframe

I would like to save the content of a spark dataframe into a csv file in s3 bucket: df_country.repartition(1).write.csv('s3n://bucket/test/csv/a',sep=",",header=True,mode='overwrite') the problem ...
0
votes
0answers
15 views

ReactJS app displaying route not found after deployment

I am tasked with deploying a ReactJS application on an AWS S3 bucket. This ReactJS application will call API endpoints that are fronted by an Apache reverse proxy SSL terminator stack. I have ...
0
votes
1answer
17 views

Requests + smart_open painfully slow

I'm using Python's requests with smart_open to download a file (a .tif file, in case that helps), and upload it to an S3 bucket without saving any temporary file. I'm looping through a few thousands ...
0
votes
0answers
16 views

Why does S3 upload return different events for Firefox vs. Chrome?

I am uploading a file to S3 using a presigned URL. I'm constructing my PUT request in typescript like this: const headers = new HttpHeaders({'Content-Type': 'text/csv'}); return new HttpRequest('PUT'...
1
vote
1answer
38 views

AWS S3 using client side enctyption

I want to implement a client side encryption for data to be uploaded to an S3 bucket. However, since I have only basic knowledge in encryption I am a little confused with the examples I found to solve ...
1
vote
0answers
17 views

How to upload image in s3 using signed url and ionic camera plugin in Ionic 4

I am using ionic native plugin for image upload with s3 signed url. camera plugin generate the base64 format image so i am not able to upload in right format takePhoto() { const options: ...
0
votes
1answer
25 views

AWS static site endpoint not loading correctly

I followed quite a complicated AWS guide on setting a static hosting for my website, and it’s currently not loading, not sure why. What I have done: Created S3 bucket for the contents of the site ...
0
votes
1answer
14 views

get audio file object from amazon s3

I want to get audio tags from an audio file. For that, I use eyed3 plugin. import eyed3 mp3_file = "The_File_Path" audiofile = eyed3.load(mp3_file) year = audiofile.tag.getBestDate() But I have only ...
-1
votes
0answers
16 views

AWS S3 bucket setup for web apps

I need to setup S3 bucket to host videos of the web app I have tried to set it up but it's not working, I have done the basic setup of s3 and getting url's but not able to get content from it
0
votes
0answers
12 views

How to check the response or status code in angular 8 for upload file to S3 Presigned URL and statusCode is 200

Request to upload file: this.httpClient.put(preSingedUrl, file, httpOptions) .subscribe((response: HttpResponse<any>) => { console.log(response); // it's returning null }); response is ...
0
votes
0answers
15 views

How to access aws s3 private bucket object from angular or node?

My amazon bucket is private so how to make authorised URL for accessing private object using angular or node?
0
votes
1answer
4 views

Load a NetworkX gpickle from S3 into AWS lambda

As the title says, I am trying to use Amazon lambda to load a NetworkX graph from S3 and I would like to use a gpickle because it is a much smaller file size than the JSON. I was able to load the ...
0
votes
1answer
15 views

How to use curl with S3 protocol to upload object swift openstack?

I want to upload file to Swift Openstack using s3 protocol with curl #!/bin/bash file=README.md bucket=bucket resource="/${bucket}/${file}" contentType="application/x-compressed-tar" dateValue=`date -...
1
vote
1answer
39 views

Creating presigned url for a S3 folder in python

I'm trying to generate a presigned url to a S3 folder (which itself contains more folders/files) and distribute it between my clients so they can download its content. i.e. by clicking the link, the ...
-1
votes
0answers
8 views

How to give access to Presto, Hive and S3 using some API key instead of username and password? [on hold]

I have access to Presto and Hive as they are in the same VPC and access to S3 using aws secret and access keys. I realised that all my clients cannot be in the same VPC(not smart enough) and that I ...
0
votes
1answer
40 views

AWS policy to limit S3 data size per IAM user

I am trying to put a restriction on how much data a given user can upload to an S3 bucket based on account tiers. For example: Tier 1: user can upload max 1GB to S3 | Tier 2: “” 5GB | Tier 3: “” 25GB ...
0
votes
2answers
23 views

How do I get file url after put file to amazon s3 in golang?

File is uploaded successfully, and I want to return file url after uploading. _, err := s3.New(s).PutObject(&s3.PutObjectInput{ Bucket: aws.String("bucket"), Key: ...
0
votes
0answers
27 views

412 (Precondition Failed) when trying to POST PDF file to S3 pre-signed URL

This is Part Two of my journey to try and upload a file from the browser to an S3 bucket. Check out Part One. What I'm Trying to Do Upload a file (PDF) from my browser client (Vue/Nuxt app) to an S3 ...
0
votes
0answers
9 views

How to load a .hdf5 model in S3 to EC2 directly?

I used keras created a .hdf5 model and stored it in S3. Now I want to load this model from S3 to EC2 directly? I wonder what is the best practice to do this? I mean I can aws s3 cp to move the model ...
-1
votes
1answer
21 views

AWS S3 CLI ls list directories without prefix into an array (and where clause)

From EC2 console terminal, I am trying to list a part of an S3 bucket directory names into an array excluding a prefix "date=" but cannot figure out a complete solution. I've already tried the ...
1
vote
0answers
37 views

How to migrate an image file from SQL Server varbinary(max) to S3 .jpg or .bmp with only AWS Lambda .Net Core

I am attempting to migrate image data from a SQL Server VARBINARY(MAX) column into AWS S3 as a .jpg via an AWS Lambda function using C#. Is this possible and how? (I have run into several dead-ends) ...
0
votes
1answer
29 views

AWS Glue job to convert table to Parquet w/o needing another crawler

Is it possible to have a Glue job re-classify a JSON table as Parquet instead of needing another crawler to crawl the Parquet files? Current set up: JSON files in partitioned S3 bucket are crawled ...