I’m contributing to a secure cloud project (well, it’s not that secure yet, but getting there..). It’s backend storage options include S3 so I want to test the S3-functionality against a locally installed S3 server.
I first tried to utilize OpenStack Object Storage (Swift) or Riak, but both solutions were rather heavy-weight and cumbersome to setup. Bear in mind, that I just wanted some fake S3 storage server which would be deployed within a local network (without any internet connection). So security, authentication, performance was mostly moot.
Then I came unto FakeS3. This is a simple Ruby gem which emulates an S3 server. Coming from a RoR world this seemed to be a perfect fit for me.
Setting up FakeS3
Being a ruby gem setup was rather easy for me (as I already had ruby installed on my computer):
1 2 3 4 5 6 |
[~]$ gem install fakes3 Fetching: fakes3-0.1.5.gem (100%) Successfully installed fakes3-0.1.5 Parsing documentation for fakes3-0.1.5 Installing ri documentation for fakes3-0.1.5 1 gem installed |
That’s it. There’s no user authentication so setup is rather overhead.. minimal. FakeS3 uses a normal file system directory for object storage – no need to setup a database.
Starting FakeS3
To start fakes3 you mostly start it while passing a storage root directory and a network port: fakes3 --root=/var/cache/fakes3 --port=10453
To make usage a bit easier I’ve added a simple init script. You can copy this into your /etc/init.d
scripts directory and should be able to use it as any other SysV init script (but beware, I’ve only tested it with Debian).
Using JetS3 Java library
First of all we need some fake host entries so that our java library will connect to the right host name. Please make sure that those host entries are setup on the fakes3 server as well as on the client computers.
To achieve this I’ve just added new /etc/hosts
entries – if you are using a local DNS server I would add those entries at the DNS server. Example of /etc/hosts
entries:
1
|
10.0.0.1 fakes3.local testme.snikt.net.fakes3.local |
So the S3 server name is fakes3.local
and the bucket id is testme.snikt.net
. How to use it from within java?
|
|
Using clients with FakeS3
s3cmd needs a special configuration file for usage with FakeS3. I’m using a slightly adapted one from the project’s github repository:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
[default] access_key = abc acl_public = False bucket_location = US cloudfront_host = cloudfront.amazonaws.com cloudfront_resource = /2008-06-30/distribution default_mime_type = binary/octet-stream delete_removed = False dry_run = False encoding = UTF-8 encrypt = False force = False get_continue = False gpg_command = None gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s gpg_passphrase = guess_mime_type = True host_base = fakes3.local:10453 host_bucket = %(bucket)s.fakes3.local:10453 human_readable_sizes = False list_md5 = False preserve_attrs = True progress_meter = True proxy_host = proxy_port = 0 recursive = False recv_chunk = 4096 secret_key = def send_chunk = 4096 simpledb_host = sdb.amazonaws.com skip_existing = False use_https = False verbosity = WARNING |
Please note the %(bucket)s.snikt.local
entry. Now you can use s3cmd
as ususal:
1
|
s3cmd --config=/home/andy/example_configuration_file la |