How to use FakeS3 for S3 testing

linux > tech
Comments

I’m contributing to a secure cloud project (well, it’s not that secure yet, but getting there..). It’s backend storage options include S3 so I want to test the S3-functionality against a locally installed S3 server.

I first tried to utilize OpenStack Object Storage (Swift) or Riak, but both solutions were rather heavy-weight and cumbersome to setup. Bear in mind, that I just wanted some fake S3 storage server which would be deployed within a local network (without any internet connection). So security, authentication, performance was mostly moot.

Then I came unto FakeS3. This is a simple Ruby gem which emulates an S3 server. Coming from a RoR world this seemed to be a perfect fit for me.

Setting up FakeS3

Being a ruby gem setup was rather easy for me (as I already had ruby installed on my computer):

1
2
3
4
5
6

[~]$ gem install fakes3
Fetching: fakes3-0.1.5.gem (100%)
Successfully installed fakes3-0.1.5
Parsing documentation for fakes3-0.1.5
Installing ri documentation for fakes3-0.1.5
1 gem installed

That’s it. There’s no user authentication so setup is rather overhead.. minimal. FakeS3 uses a normal file system directory for object storage – no need to setup a database.

Starting FakeS3

To start fakes3 you mostly start it while passing a storage root directory and a network port: fakes3 --root=/var/cache/fakes3 --port=10453

To make usage a bit easier I’ve added a simple init script. You can copy this into your /etc/init.d scripts directory and should be able to use it as any other SysV init script (but beware, I’ve only tested it with Debian).

Using JetS3 Java library

First of all we need some fake host entries so that our java library will connect to the right host name. Please make sure that those host entries are setup on the fakes3 server as well as on the client computers.

To achieve this I’ve just added new /etc/hosts entries – if you are using a local DNS server I would add those entries at the DNS server. Example of /etc/hosts entries:

1

10.0.0.1      fakes3.local testme.snikt.net.fakes3.local

So the S3 server name is fakes3.local and the bucket id is testme.snikt.net. How to use it from within java?

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

AWSCredentials awsCredentials = new AWSCredentials("don't need an access key", "don't need a secret key"); 
S3Service s3service = new RestS3Service(awsCredentials);

/* overwrite server information */
Jets3tProperties props = s3service.getJetS3tProperties();
props.setProperty("s3service.s3-endpoint-http-port", "10453");
props.setProperty("s3service.https-only", "false");
props.setProperty("s3service.s3-endpoint", "fakes3.local");

/* connect to the face s3 server */
CredentialsProvider credentials = s3service.getCredentialsProvider();
s3service = new RestS3Service(awsCredentials, "test-app", credentials , props);

/* connect or create a new s3bucket */
S3Bucket s3bucket = s3service.getBucket("testme.snikt.net");
if(s3bucket == null) {
        s3bucket = s3service.createBucket(bucketId);
}

/* store something on the server */
byte[] binary_blob = [...];
StorageObject obj = new S3Object("some-id", binary_blob);
s3service.putObject(s3bucket.getNmae(), obj);

Using clients with FakeS3

s3cmd needs a special configuration file for usage with FakeS3. I’m using a slightly adapted one from the project’s github repository:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

[default]
access_key = abc
acl_public = False
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2008-06-30/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
force = False
get_continue = False
gpg_command = None
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase = 
guess_mime_type = True
host_base = fakes3.local:10453
host_bucket = %(bucket)s.fakes3.local:10453
human_readable_sizes = False
list_md5 = False
preserve_attrs = True
progress_meter = True
proxy_host = 
proxy_port = 0
recursive = False
recv_chunk = 4096
secret_key = def
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
use_https = False
verbosity = WARNING

Please note the %(bucket)s.snikt.local entry. Now you can use s3cmd as ususal:

1

s3cmd --config=/home/andy/example_configuration_file la
comments powered by Disqus