AWS Lambda to extract static assets from CodePipeline Artifact to S3
€30-250 EUR
Cerrado
Publicado hace casi 7 años
€30-250 EUR
Pagado a la entrega
Dear freelancers,
we need a simple AWS-Lambda function that may be attached to AWS-CodePipeline.
The CodePipeline-artifact passed to the Lambda-function is round about 200 MB to 300 MB in size with about 10.000 assets that need to be extracted to S3.
Only files with definable file-extensions within definable folders and their subfolders should be extracted and synced with an S3-bucket. Therefore, an JSON-String will be set at CodePipeline invocation of Lambda-Function.
*Parameters that should be set in CodePipeline:*
- Folders (array of Strings containing foldernames, each with trailing slash)
- Fileextensions (array of Strings containing f.e. ".js", ".css" and so on)
- Target-Bucket (string - may be in another region as the codepipeline)
*Rules*
- files outside the defined folders should be ignored.
- files with other file extensions should be ignored.
- the file-extension must exactly match - while *.js is allowed, *.json might not. *.[login to view URL] would be okay, if *.js is allowed.
- files existing on S3 (that match the rules for folder and file-type above) but do not exist in the Artifact should be removed from S3 (they may have existed in a previous version).
- Only changed files should be copied to S3 (because of the amount of assets)
- The folders in S3 may contain more file / objects then AWS API command listObjects() will return in a single request
*Performance / Speed is really important.*
- The whole process should take as less time as possible best much below 1 minute.
- The listObjects() command should only be executed against the listed folders because the whole bucket may contain 250k files and more in folders, that are irrelevant for the sync-process.
- The sync should be processed async so that putObjects & deleteObjects operations are processed in parallel but the number of parallel putObject and deleteObject operations should be capped so that they don't exceed AWS limits or result in a out of memory exception.
- On production, the lambda-function will get max Memory possible assigned.
Node.JS ore Python may be used (if possible with one of them) to develop this functionality.
If you have any questions, feel free to ask.
Hi
I really don't recommend to add this much heavy processing in Lambda, as it'll definitely timeout even if you give it maximum execution time of 5 min.
Instead I'd want to know the entire pipeline process, so that I can propose a better solution for your use case.
BTW I'm a super expert in AWS.
Ranked #1 in AWS Freelancer Category.
You can have a look at my profile to view my past AWS experience.
https://www.freelancer.com/u/joystick220.html
Thanks
Joy
Hi sir,
I am scraping expert, I have did more than 350+ scraping project, please check my feedback then you will know.
Can we discuss more details about this project? then I will provide example data/script for you.
Thanks,
Kimi
Hey there,
I have extensive experience of writing Lambda functions using Nodejs/Python and Java. I've also configured Lambda to use PHP as well, so I'm very much well versed in developing Lambda functions. You can confirm this by checking out my profile page, you will see lots of Lambda related projects.
I've worked with almost all of the Amazonn AWS services as well including DynamoDB, CodeDeploy, ElasticBeanstalk, Elastic Load Balancer, etc.
Please feel free to ask if you've any questions.
Thanks
Hi there,
First point I would like to make is your requirement of less than 1 minute upload time is not practical / technically feasible. Apart from this particular requirement I will give you a lambda function written with Python that will satisfy all other requirements.
we can start ur work immediately.
we have 10 years experience in coding and development.
we are expert in writing programs in JAVA, .NET, C, C++,C#
we have experience in hadoop,python coding,raspberry Pi
django
PHP
Arduino
Magento
metatrader
web design
HTML,HTML5
graphic design
wordpress
Android
unity3d