{"id":428,"date":"2021-11-26T22:08:53","date_gmt":"2021-11-27T03:08:53","guid":{"rendered":"https:\/\/wangxiaohu.com\/blog\/?p=428"},"modified":"2021-11-26T23:10:20","modified_gmt":"2021-11-27T04:10:20","slug":"setting-up-vpc-endpoint-for-amazon-s3","status":"publish","type":"post","link":"https:\/\/wangxiaohu.com\/blog\/?p=428","title":{"rendered":"Setting up VPC endpoint for Amazon S3"},"content":{"rendered":"\n<p>By default, accessing S3 resources from any instance or Kubernetes pod within a VPC involves outbound traffic via NAT or IGW. Not only this is less efficient, it also incurs a service fee due to the traffic. The cost can be significant if traffic is huge.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Setup<\/h2>\n\n\n\n<p>To keep the traffic within VPC, an S3 accesspoint for the specific S3 resources and a VPC endpoint can be created by following <a href=\"https:\/\/aws.amazon.com\/blogs\/storage\/managing-amazon-s3-access-with-vpc-endpoints-and-s3-access-points\/\">the general instruction<\/a>.<\/p>\n\n\n\n<p>After that, double check the policies at two places.<\/p>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"1.-Policy-for-IAM\">1. Policy for IAM<\/h5>\n\n\n\n<p>Go to <strong>AWS Console<\/strong> \u2192 <strong>IAM <\/strong>\u2192 <strong>User<\/strong>. It is supposed to be a service account that has following policy attached<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n    \"Version\": \"2012-10-17\",\n    \"Statement\": &#91;\n        {\n            \"Sid\": \"VisualEditor0\",\n            \"Effect\": \"Allow\",\n            \"Action\": &#91;\n                \"s3:ReplicateObject\",\n                \"s3:PutObject\",\n                \"s3:GetObject\",\n                \"s3:ListBucket\",\n                \"s3:DeleteObject\"\n            ],\n            \"Resource\": \"*\"\n        }\n    ]\n}<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-verse\">As a security best practice, DO NOT grant <code>Allow<\/code> to all actions, i.e. <code>\"Action\": [\"*\"]<\/code>.<\/pre>\n\n\n\n<h5 class=\"wp-block-heading\" id=\"2.-Policy-for-the-endpoint\">2. Policy for the endpoint<\/h5>\n\n\n\n<p>Go to <strong>AWS Console<\/strong> \u2192 <strong>VPC <\/strong>\u2192 <strong>Endpoints <\/strong>\u2192 <strong>the endpoint<\/strong> \u2192 <strong>Policy<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n    \"Version\": \"2012-10-17\",\n    \"Id\": \"Policy1637977229005\",\n    \"Statement\": &#91;\n        {\n            \"Sid\": \"Stmt1637977226759\",\n            \"Effect\": \"Allow\",\n            \"Principal\": \"*\",\n            \"Action\": \"s3:*\",\n            \"Resource\": \"*\"\n        }\n    ]\n}<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-verse\">This endpoint policy is less constrained than the policy for user, because want the policy for user to be more specific.<\/pre>\n\n\n\n<p>After all, run <code>traceroute s3.ap-southeast-1.amazonaws.com<\/code> from any of the instances within the VPC to verify.<\/p>\n\n\n\n<ul><li>Following output shows it is successful<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">$ traceroute s3.ap-southeast-1.amazonaws.com\ntraceroute to s3.ap-southeast-1.amazonaws.com (52.219.129.42), 30 hops max, 60 byte packets\n 1  * * *\n 2  * * *\n 3  * * *\n    ...\n28  * * *\n29  * * *\n30  * * *<\/pre>\n\n\n\n<ul><li>Following output shows it is still going thru NAT\/IGW<\/li><\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">$ traceroute s3.ap-southeast-1.amazonaws.com\ntraceroute to s3.ap-southeast-1.amazonaws.com (52.219.40.74), 30 hops max, 60 byte packets\n 1  ec2-175-41-128-191.ap-southeast-1.compute.amazonaws.com (175.41.128.191)  42.290 ms ec2-175-41-128-195.ap-southeast-1.compute.amazonaws.com (175.41.128.195)  3.103 ms ec2-18-141-171-23.ap-southeast-1.compute.amazonaws.com (18.141.171.23)  36.088 ms\n 2  100.65.32.176 (100.65.32.176)  1.149 ms 100.65.32.160 (100.65.32.160)  1.132 ms 100.65.33.144 (100.65.33.144)  4.671 ms\n 3  100.66.16.248 (100.66.16.248)  3.400 ms 100.66.16.26 (100.66.16.26)  8.787 ms 100.66.16.244 (100.66.16.244)  4.695 ms\n 4  100.66.19.122 (100.66.19.122)  3.511 ms 100.66.19.204 (100.66.19.204)  17.491 ms 100.66.18.104 (100.66.18.104)  18.193 ms\n 5  100.66.3.241 (100.66.3.241)  15.377 ms 100.66.3.137 (100.66.3.137)  154.101 ms 100.66.3.61 (100.66.3.61)  19.897 ms\n 6  100.66.0.135 (100.66.0.135)  10.907 ms 100.66.0.165 (100.66.0.165)  9.064 ms 100.66.0.201 (100.66.0.201)  11.826 ms\n 7  100.65.2.41 (100.65.2.41)  3.090 ms 100.65.3.41 (100.65.3.41)  2.821 ms 100.65.2.41 (100.65.2.41)  2.648 ms\n 8  s3-ap-southeast-1.amazonaws.com (52.219.40.74)  0.518 ms  0.569 ms  0.578 ms<\/pre>\n\n\n\n<p>Follow the <a href=\"https:\/\/aws.amazon.com\/premiumsupport\/knowledge-center\/connect-s3-vpc-endpoint\/\">troubleshooting steps<\/a> if anything does not work.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Usage<\/h2>\n\n\n\n<p>Now we have both S3 access point and the VPC endpoint for S3 setup.<\/p>\n\n\n\n<p>To access the S3 resource within VPC, instead of using <code>s3:\/\/&lt;bucket name&gt;<\/code> directly, use<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\"><code>s3:\/\/arn:aws:s3:ap-southeast-1:&lt;account number&gt;:accesspoint\/&lt;bucket name&gt;<\/code><\/pre>\n\n\n\n<p>Some 3rd party library built for AWS S3 might not recognize the S3 access point URL and throw exceptions like<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Caused by: java.lang.NullPointerException: null uri host.\n        at java.util.Objects.requireNonNull(Objects.java:228)\n        at org.apache.hadoop.fs.s3native.S3xLoginHelper.buildFSURI(S3xLoginHelper.java:71)\n        at org.apache.hadoop.fs.s3a.S3AFileSystem.setUri(S3AFileSystem.java:486)\n        at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:246)\n        at org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:123)\n        ... 24 more<\/code><\/pre>\n\n\n\n<p>In that case, <a href=\"https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/userguide\/access-points-alias.html\">S3 access point alias<\/a> could be used. It can be found under <strong>S3 Bucket<\/strong> \u2192 <strong>Access Points<\/strong>. It looks something like<\/p>\n\n\n\n<p><code>s3:\/\/&lt;bucket name>-1ks47nsk5hyxi845kebsen1nyf11caps1b-s3alias<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>By default, accessing S3 resources from any instance or &hellip; <a href=\"https:\/\/wangxiaohu.com\/blog\/?p=428\" class=\"more-link\">\u7ee7\u7eed\u9605\u8bfb<span class=\"screen-reader-text\">Setting up VPC endpoint for Amazon S3<\/span><\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","_jetpack_memberships_contains_paid_content":false},"categories":[1],"tags":[51],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack-related-posts":[],"jetpack_shortlink":"https:\/\/wp.me\/pdhZ2A-6U","_links":{"self":[{"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/428"}],"collection":[{"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=428"}],"version-history":[{"count":2,"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/428\/revisions"}],"predecessor-version":[{"id":434,"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/428\/revisions\/434"}],"wp:attachment":[{"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=428"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=428"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wangxiaohu.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=428"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}