Guidance on Shared Storage for AWS

[Rees1993]

I'm submitting a...

• bug report
• feature request
• general question

Description

My workplace is using Imager-X on numerous sites and have recently moved to AWS Fargate (the hosting company we use recommended and set it up). However, we have ran into this issue whereupon deployment, the local imager cache gets removed.

I see in the docs FAQS that you mention that this should be on shared storage but I am currently unsure how to do this...(new to Craft CMS & AWS)... I was wondering if you have ever had experience using AWS for this and got any guidance in how to get this working? I understand that this more of an AWS/Imager-X setup question so any help is appreciated.

Below is what we currently have in the imager-x.php file:

return [
    '*' => [
        'imagerSystemPath' => realpath(__DIR__ . '/../web/images/imager') . '/',
        'cacheEnabled' => true,
        'cacheDuration' => 60 * 60 * 24 * 30, // 30 days
        'cacheDurationRemoteFiles' => 60 * 60 * 24 * 30, // 30 days
        'allowUpscale' => false,
        'resizeFilter' => 'mitchell',
        'useCwebp' => true,
        'cwebpPath' => '/usr/local/bin/cwebp',
        'storages' => [],
        'storageConfig' => [
            'aws' => [
                'accessKey' => getenv('AWS_ACCESS_KEY_ID'),
                'secretAccessKey' => getenv('AWS_SECRET_ACCESS_KEY'),
                'bucket' => getenv('BUCKET'),
                'region' => getenv('REGION'),
                'folder' => 'transforms',
                'public' => true,
                'requestHeaders' => array(),
                'storageType' => 'standard', // 'standard' or 'rrs' (reduced redundancy storage)
                'cloudfrontInvalidateEnabled' => false,
                'cloudfrontDistributionId' => '',
            ],
        ],
    ],
    'dev' => [
        'imagerUrl' => '/images/imager/', //Local
        'storages' => [],
    ],
    'staging' => [
        'imagerUrl' => 'https://s3.eu-west-1.amazonaws.com/static.laura-wall.btb-staging.com/transforms/', //AWS
        'storages' => ['aws'],
    ],
];

Additional info

• Imager version & edition: 3.4.0
• Imager transformer: Craft (Default)
• Craft version: 3.6.12
• PHP version: 7.4.12
• Image driver & version: GD 7.4.12
• Database driver & version: MySQL 5.7
• Plugins & versions (if relevant): Pro

[aelvan]

Hi,

Unfortunately I don't have much experience with the AWS eco-system beyond S3 and CloudFront, so I can't comment specifically on AWS Fargate, and what's needed. But in general, you'd want the imagerSystemPath pointed to some shared volume that you mount/symlink as part of your deploy routine. You should also do the same for the craft storage path, as outline here: https://craftcms.com/knowledge-base/configuring-load-balanced-environments#storage-folder

For details on how to do this in AWS Fargate, I suggest asking in the #devops channel in the Craft Discord. There're so many helpful people hanging out there, with experience in most things internet. :)

Let me know if you need any more help.

[lpearson-trek10]

I am not using Imager-x but rather Imager (I know a little behind the times) but we are also on Fargate and have the exact same issue with local cache. I have not really looked too deep into it (just not that big of a deal for us yet) but what I plan on looking into when I get to it is using EFS. As for April 2020 mounting EFS on Fargate has been supported and it seems like a viable option for your problem (as well as mine). Would love to hear more if you get a chance to look into that as a solution!

[JoshCoady]

Each fargate container has its own local storage. Typically a deploy destroys the existing containers and boots up new ones (that have the new code). This also wipes out the local storage on the old ones. To make some storage persistent, you need to configure your fargate containers with shared volumes (normally backed by EFS). Then configure imagerSystemPath to point to a dir on that shared volume. This will make the imager cache persistent and it will also make it shared among all containers (if you are running multiple concurrently). https://aws.amazon.com/premiumsupport/knowledge-center/ecs-fargate-mount-efs-containers-tasks/

[noahjahn]

I'm hosting a craft site in docker containers on a platform that provides autoscaling, as long as I don't need to persist or share data on the local file system of the containers.

Is there any chance that the imagerSystemPath could store its cache in an external service like Redis or maybe AWS S3?

[mattbloomfield]

I have a similar setup using Platform.sh and we just use a mount. It works nicely.

[noahjahn]

I'm hosting a craft site in docker containers on a platform that provides autoscaling, as long as I don't need to persist or share data on the local file system of the containers.

Is there any chance that the imagerSystemPath could store its cache in an external service like Redis or maybe AWS S3?

The platform we're using is Render

[aelvan]

@noahjahn I don't think either AWS S3 or Redis would be good options for this. Checking S3 for every single transform would introduce insane amounts of latency. I guess Redis could work if someone configured it to be persistent, and made sure that cached items wasn't evicted, which in my experience can be pretty hard.

I think the only viable option would be to use the database, like Craft does for its transforms. I'm contemplating adding an option for this for the next major release, which will probably be synchronised with Craft 4.0.

[lpearson-trek10]

Maybe this is getting too far out of scope for what is being asked here but as far as transforms on an s3 objects go AWS has a clear and clean possible solution https://docs.aws.amazon.com/AmazonS3/latest/userguide/transforming-objects.html

They even has a blog on doing this exact thing:
https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and-amazon-api-gateway/

Would be pretty awesome to be able to deploy a lambda function that could use imager somehow in it.

So the idea is that this would store the original asset in S3 then make additions to the url that the transformer could understand then I can just cache the image and redirect to that and serve it straight from S3. That takes load off my server and onto AWS infrastructure just where I like it :)

[aelvan]

This is a bit besides the issue that were raised here, but offloading the transforms to a service that does that is absolutely an avenue worth exploring for people that can't have a persistent cache.

Imager X has support for several such services already, Imgix is built-in and integrations with ImageBoss and AWS Serverless Image Transformer is provided as first-party addons. As far as I can tell (which is never easy with AWS), the Serverless Image Handler is just a more fleshed out solution of what's described in the blog post you linked to. It's built on Sharp, and has a few short-comings compared to the local transformer and Imgix, but is a very cheap and easy alternative.

[noahjahn]

@noahjahn I don't think either AWS S3 or Redis would be good options for this. Checking S3 for every single transform would introduce insane amounts of latency. I guess Redis could work if someone configured it to be persistent, and made sure that cached items wasn't evicted, which in my experience can be pretty hard.

I think the only viable option would be to use the database, like Craft does for its transforms. I'm contemplating adding an option for this for the next major release, which will probably be synchronised with Craft 4.0.

Thank you for the reply!

You're definitely right, S3 would introduce too much latency.

I still think Redis would be a good alternative even to the database (maybe there is a choice on which service to use). For my specific use case, I already have a Redis service running in Render that stays persisted just like the database.

[aelvan]

After giving this a bit more thought, it occurred to me that it would probably be best to build this on top of the Yii2 data cache, and provide a way to configure it to use the cache components that already exists. That would make it possible to cache with both the database, and something like redis or memcache.

Also, as noted in another comment, if you have the assets on S3 and only need to transform those (ie, no external images for instance), you could consider setting up a Serverless Image Handler and using the AWS Serverless Transformer to do the transforms instead of doing it on the server. Or, use Imgix or ImageBoss, which both will offload the transforms from the server, and make the local cache redundant.

[Rees1993]

I am no longer using Imager-X (sorry @aelvan) but wondered if this would be better off as a discussion than an issue? I am unsure if this is possible though

[aelvan]

Good idea, done. 👍