As part of our ongoing effort to improve archive search performance and efficiency, we are introducing changes to the S3 bucket permissions needed for the logs/Traces S3 bucket. These changes will allow us to compact log and trace archive files, resulting faster query times and slightly smaller storage footprint. The update will take effect on October 15th, 2024.
Who is this relevant for? This change applies to any Coralogix customer that uses Coralogix’s archive feature for logs and/or traces stored in S3. Since this is one of the core components in Coralogix most customers will need to update the permissions.
What is happening? On October 15th, 2024, Coralogix will introduce a new compaction method for the archive files for Logs and Traces stored in S3 bucket. Coralogix will need new permissions to enable file compaction.
What should you do? To benefit from these improvements, please update your S3 bucket permissions by October 15th, 2024. Add the following permissions to the S3 bucket: s3:DeleteObject, s3:DeleteObjectVersion, s3:DeleteObjectTagging, s3:DeleteObjectVersionTagging We’ve updated our CloudFormation templates and tutorials to assist you in making these changes.
Why do you need all those permissions? s3:DeleteObject – Allows deleting objects from the S3 bucket. s3:DeleteObjectVersion – Allows deleting all versions of an object when bucket versioning is enabled. Object cannot be deleted when they have versions. s3:DeleteObjectTagging – Allows deleting tags from an object (used by the Archive retention feature). Object cannot be deleted when they are tagged. s3:DeleteObjectVersionTagging – Enables deleting tags for a versioned object.
Why is this change being made? This update is designed to enhance both storage efficiency and system performance:
Important for ETL users: If you run ETL processes on your S3 bucket, note that during compaction, both the pre-compacted and compacted log files might exist momentarily. This could briefly cause duplicate data in your ETL workflow. Please update your processes to account for this.
What happens if you don’t update? If the S3 permissions are not updated, we will implement a workaround that replaces compacted log files with zero-byte files in your bucket. Although this workaround reduces storage, it is less efficient and may incur minor costs related to PUT objects.