aws-sdk-cpp part 1 - Alien TechLAB

Posted by admin on December 26, 2016February 24, 2017

Introduction

In this post, I want to present how to install aws-sdk-cpp on Linux Mint 18 and how to write basic code to work with buckets and objects.

Preparation

Software I used:
1. Virtual Box
2. Linux Mint 18 “Sahra” 64 bit
3. CloudBerry Explorer

Create new virtual machine on Virtual Box and install Linux Mint 18 on it. New Mint requires additional installations to work with aws, so create prepare.sh file and copy script below into it. Remember to set executable attribute for this file (chmod +x prepare.sh) and run this script with sudo ./prepare.sh or as root. This script will do whole job for you. But keep in mind that compiling aws-sdk-cpp takes (on my machine) 2 hours.

prepare.sh

# Run this script with sudo or as root.

apt install g++ -y

apt install git -y

apt install cmake -y

apt-get install libssl-dev -y

apt-get install libcurl4-openssl-dev -y

apt-get install uuid-dev -y

apt-get install libboost-all-dev -y

git clone https://github.com/aws/aws-sdk-cpp

mkdir build_dir

cd build_dir

cmake -DCMAKE_BUILD_TYPE=Release ../aws-sdk-cpp

make

make install

cd ..

rm -rf build_dir

rm -rf aws-sdk-cpp

Hello, world !

Now you're ready to code. Let's create "hello world". Create hello_aws.cpp file and copy code below into it. This code does nothing yet, but it's good starting point. If you can compile and run it, that means everything is installed properly.

hello_aws.cpp

#include <aws/core/Aws.h>

using namespace Aws;

int main()

{

SDKOptions options;

InitAPI(options);

// Other code.

ShutdownAPI(options);

return 0;

}

In order to build hello_aws.cpp file we need building script. Create build_and_run.sh file and copy script below into it. Again, remeber about setting executable attribute (chmod +x build_and_run.sh).

build_and_run.sh

export LD_LIBRARY_PATH="/usr/local/lib"

rm -f ./a.out

g++ hello_aws.cpp -std=c++11 -laws-cpp-sdk-core -laws-cpp-sdk-s3

./a.out

AWSCredentials

Next thing we have to add is instance of AWSCredentials class. As it’s name suggests, it keeps access key and secret key. The first one is like login, the second, like password. Both are created for you on aws.amazon.com website. Code after changes will look like this:

#include <aws/core/Aws.h>

#include <aws/core/auth/AWSCredentialsProvider.h>

using namespace Aws;

using namespace Aws::Auth;

int main()

{

SDKOptions options;

InitAPI(options);

AWSCredentials creadentials;

creadentials.SetAWSAccessKeyId("Put here your own access key.");

creadentials.SetAWSSecretKey("Put here your own secret key.");

ShutdownAPI(options);

return 0;

}

DefaultRetryStrategy

When we are dealing with network, sooner or later something goes wrong. Connection is lost, etc. In such case you should try couple times. You should retry.

To decide how many times application should try again and how long it should wait between retries, we need retry policy. AWS provides two classes to address this issue: RetryStrategy and DefaultRetryStrategy. The first one is abstract class, the later is default implementation that derives from RetryStrategy. I’m going to use the default retry policy because it’s enough for me. What you should know is that, if you need custom retry policy, you can create your own class that derives from RetryStrategy and then override methods: ShouldRetry and CalculateDelayBeforeNextRetry. You can find more details here.

Because ClientConfiguration takes std::shared_ptr, I created DefaultRetryStrategy using new operator and then used that pointer (defaultRetryStrategy) to create retryStrategy std::shared_ptr.

#include <aws/core/Aws.h>

#include <aws/core/auth/AWSCredentialsProvider.h>

#include <aws/core/client/DefaultRetryStrategy.h>

using namespace std;

using namespace Aws;

using namespace Aws::Auth;

using namespace Aws::Client;

int main()

{

SDKOptions options;

InitAPI(options);

AWSCredentials creadentials;

creadentials.SetAWSAccessKeyId("Put here your own access key.");

creadentials.SetAWSSecretKey("Put here your own secret key.");

RetryStrategy* defaultRetryStrategy;

defaultRetryStrategy = new DefaultRetryStrategy();

shared_ptr<RetryStrategy> retryStrategy(defaultRetryStrategy);

ShutdownAPI(options);

return 0;

}

ClientConfiguration

Since we have std::shared_ptr to our retry strategy, we can create ClientConfiguration object. As it’s name suggests it contains settings for S3Client.

#include <aws/core/Aws.h>

#include <aws/core/auth/AWSCredentialsProvider.h>

#include <aws/core/client/DefaultRetryStrategy.h>

#include <aws/s3/S3Client.h>

using namespace std;

using namespace Aws;

using namespace Aws::Auth;

using namespace Aws::Client;

int main()

{

SDKOptions options;

InitAPI(options);

AWSCredentials creadentials;

creadentials.SetAWSAccessKeyId("Put here your own access key.");

creadentials.SetAWSSecretKey("Put here your own secret key.");

RetryStrategy* defaultRetryStrategy;

defaultRetryStrategy = new DefaultRetryStrategy();

shared_ptr<RetryStrategy> retryStrategy(defaultRetryStrategy);

ClientConfiguration configuration;

configuration.scheme = Http::Scheme::HTTPS;

configuration.connectTimeoutMs = 0;

configuration.requestTimeoutMs = 0;

configuration.retryStrategy = retryStrategy;

ShutdownAPI(options);

return 0;

}

Timeouts are set to zero (it means infinity) to avoid timeout errors. It’s problematic when you are just starting your adventure with AWS. To use ClientConfiguration class we need to include S3Client.h file.

S3Client

S3Client object is essential for our application. It takes part in every operation like: creating bucket, uploading file, etc.
It’s worth to keep in mind that S3Client class is thread safe, so it’s instance can be shared by many threads and it has tcp connection pool. This pool is by default set to 25 connections but it can be set up in ClientConfiguration.maxConnections field.

#include <aws/core/Aws.h>

#include <aws/core/auth/AWSCredentialsProvider.h>

#include <aws/core/client/DefaultRetryStrategy.h>

#include <aws/s3/S3Client.h>

using namespace std;

using namespace Aws;

using namespace Aws::Auth;

using namespace Aws::Client;

using namespace Aws::S3;

int main()

{

SDKOptions options;

InitAPI(options);

AWSCredentials creadentials;

creadentials.SetAWSAccessKeyId("Put here your own access key.");

creadentials.SetAWSSecretKey("Put here your own secret key.");

RetryStrategy* defaultRetryStrategy;

defaultRetryStrategy = new DefaultRetryStrategy();

shared_ptr<RetryStrategy> retryStrategy(defaultRetryStrategy);

ClientConfiguration configuration;

configuration.scheme = Http::Scheme::HTTPS;

configuration.connectTimeoutMs = 0;

configuration.requestTimeoutMs = 0;

configuration.retryStrategy = retryStrategy;

S3Client* client = new S3Client(creadentials, configuration);

delete client;

ShutdownAPI(options);

return 0;

}

Important!
It’s very important to delete client object before call to ShutdownAPI function. Otherwise you’ll get Segmentation fault error. For that reason you shouldn’t instantiate client as local object like this:

1	S3Client client(creadentials, configuration);

because local object will be destroyed after main function returns (so after ShutdownAPI is called).

Let’s create bucket

Almost all operations follow the same pattern.
1. Create request object.
2. Pass the request object to client’s request method.
3. Request method returns outcome object.
4. The outcome object tells you if the request was successful with “IsSuccess” method.
5. If successful, you can get the result object from outcome object with “GetResult” method.
6. If not successful, you can get error object from outcome object with “GetError” method.

To create bucket, I wrote CreateBucket function that takes, previously created, client from main function and bucket’s name.

Important!
Bucket name MUST BE GLOBALLY UNIQUE! It’s like domain. You cannot create “abc” bucket, because someone in the world already has bucket with such name.

#include <aws/core/Aws.h>

#include <aws/core/auth/AWSCredentialsProvider.h>

#include <aws/core/client/DefaultRetryStrategy.h>

#include <aws/s3/S3Client.h>

#include <aws/s3/model/CreateBucketRequest.h>

using namespace std;

using namespace Aws;

using namespace Aws::Auth;

using namespace Aws::Client;

using namespace Aws::S3;

using namespace Aws::S3::Model;

void CreateBucket(const S3Client* s3Client, const string& bucketName)

{

CreateBucketRequest request;

request.SetBucket(bucketName.data());

CreateBucketOutcome outcome = s3Client->CreateBucket(request);

if(outcome.IsSuccess() == false)

{

auto error = outcome.GetError();

cout << (int)error.GetErrorType() << endl;

cout << error.GetMessage() << endl;

cout << error.GetExceptionName() << endl;

}

int main()

{

SDKOptions options;

InitAPI(options);

AWSCredentials creadentials;

creadentials.SetAWSAccessKeyId("Put here your own access key.");

creadentials.SetAWSSecretKey("Put here your own secret key.");

RetryStrategy* defaultRetryStrategy;

defaultRetryStrategy = new DefaultRetryStrategy();

shared_ptr<RetryStrategy> retryStrategy(defaultRetryStrategy);

ClientConfiguration configuration;

configuration.scheme = Http::Scheme::HTTPS;

configuration.connectTimeoutMs = 0;

configuration.requestTimeoutMs = 0;

configuration.retryStrategy = retryStrategy;

S3Client* client = new S3Client(creadentials, configuration);

CreateBucket(client, "some_globally_unique_bucket_name");

delete client;

ShutdownAPI(options);

return 0;

}

This pice of code finally does something. You can use CloudBerry Explorer to see if bucket really exists. It's usefull to have other tool to test your code at this stage.

We have bucket. Now what?

This short tutorial shows you how to setup Linux Mint and how to write very simple application that uses aws-sdk-cpp. I’m not going to talk about all details. Below you have source code that shows you how to do other things, like: delete bucket, list buckets, upload object, download object, delete object, get size of object, etc.

Here you have some useful links, especially reference manual.
Amazon S3 website
How to create account by barakpinch
Reference Manual for AWS SDK for C++

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

#include <aws/core/Aws.h>

#include <aws/core/auth/AWSCredentialsProvider.h>

#include <aws/core/client/DefaultRetryStrategy.h>

#include <aws/s3/S3Client.h>

#include <aws/s3/model/CreateBucketRequest.h>

#include <aws/s3/model/DeleteBucketRequest.h>

#include <aws/s3/model/PutObjectRequest.h>

#include <aws/s3/model/GetObjectRequest.h>

#include <aws/s3/model/DeleteObjectRequest.h>

#include <aws/s3/model/ListObjectsRequest.h>

#include <aws/s3/model/HeadObjectRequest.h>

#include <boost/iostreams/stream_buffer.hpp>

using namespace std;

using namespace Aws;

using namespace Aws::Auth;

using namespace Aws::Client;

using namespace Aws::S3;

using namespace Aws::S3::Model;

using namespace boost::iostreams;

S3Client* MyClient = NULL;

//----------------------------------------------------------------------------------------------------------------

void DisplayError(AWSError<S3Errors> error)

{

cout << (int)error.GetErrorType() << endl;

cout << error.GetMessage() << endl;

cout << error.GetExceptionName() << endl;

}

//----------------------------------------------------------------------------------------------------------------

void CreateBucket(const string& bucketName)

{

CreateBucketRequest request;

request.SetBucket(bucketName.data());

CreateBucketOutcome outcome = MyClient->CreateBucket(request);

if(outcome.IsSuccess() == false)

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void DeleteEmptyBucket(const string& bucketName)

{

DeleteBucketRequest request;

request.SetBucket(bucketName.data());

DeleteBucketOutcome outcome = MyClient->DeleteBucket(request);

if(outcome.IsSuccess() == false)

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void ListBuckets()

{

ListBucketsOutcome outcome = MyClient->ListBuckets();

if(outcome.IsSuccess() == true)

{

ListBucketsResult result = outcome.GetResult();

for(Bucket bucket : result.GetBuckets())

{

cout << bucket.GetName() << endl;

}

else

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void UploadObject(const string& bucketName, const string& objectName, const char* buffer, const int bufferSize)

{

stream_buffer<array_source> stmbuf(buffer, bufferSize);

iostream* stm = new iostream(&stmbuf);

shared_ptr<IOStream> ptr(stm);

PutObjectRequest request;

request.SetBucket(bucketName.data());

request.SetKey(objectName.data());

request.SetBody(ptr);

request.SetContentLength(bufferSize);

PutObjectOutcome outcome = MyClient->PutObject(request);

if(outcome.IsSuccess() == false)

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void DownloadObject(const string& bucketName, const string& objectName, const char* buffer, const int bufferSize)

{

GetObjectRequest request;

request.SetBucket(bucketName.data());

request.SetKey(objectName.data());

GetObjectOutcome outcome = MyClient->GetObject(request);

if(outcome.IsSuccess() == true)

{

GetObjectResult result = outcome.GetResultWithOwnership();

IOStream& body = result.GetBody();

body.read((char*)buffer, bufferSize);

}

else

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void DeleteObject(const string& bucketName, const string& objectName)

{

DeleteObjectRequest request;

request.SetBucket(bucketName.data());

request.SetKey(objectName.data());

DeleteObjectOutcome outcome = MyClient->DeleteObject(request);

if(outcome.IsSuccess() == false)

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void ListObjects(const string& bucketName)

{

ListObjectsRequest request;

request.SetBucket(bucketName.data());

// Downloades up to 1000 objects.

ListObjectsOutcome outcome = MyClient->ListObjects(request);

if(outcome.IsSuccess() == true)

{

ListObjectsResult result = outcome.GetResult();

Vector<Object> objects = result.GetContents();

for (Object object : objects)

{

cout << object.GetKey() << endl;

}

else

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void PrintObjectSize(const string& bucketName, const string& objectName)

{

HeadObjectRequest request;

request.SetBucket(bucketName.data());

request.SetKey(objectName.data());

HeadObjectOutcome outcome = MyClient->HeadObject(request);

if(outcome.IsSuccess() == true)

{

HeadObjectResult result = outcome.GetResult();

cout << (uint64_t)result.GetContentLength() << endl;

}

else

{

DisplayError(outcome.GetError());

}

//----------------------------------------------------------------------------------------------------------------

void BasicOperations()

{

string bucketName = "alientechlab-aws-test-bucket";

string objectName = "file.txt";

char uploadData[] = "This is some data to be uploaded as object.";

char downloadBuffer[1024];

// Uncomment operation you need.

//CreateBucket(bucketName);

//ListBuckets();

//UploadObject(bucketName, objectName, uploadData, sizeof(uploadData));

//DownloadObject(bucketName, objectName, downloadBuffer, sizeof(downloadBuffer));

//cout << downloadBuffer << endl;

//ListObjects(bucketName);

//PrintObjectSize(bucketName, objectName);

//DeleteObject(bucketName, objectName);

//DeleteEmptyBucket(bucketName);

}

//----------------------------------------------------------------------------------------------------------------

int main()

{

SDKOptions options;

InitAPI(options);

//..................................................................................

AWSCredentials creadentials;

creadentials.SetAWSAccessKeyId("Put here your own access key."); // It's like login

creadentials.SetAWSSecretKey("Put here your own secret key."); // It's like password

RetryStrategy* defaultRetryStrategy;

defaultRetryStrategy = new DefaultRetryStrategy();

shared_ptr<RetryStrategy> retryStrategy(defaultRetryStrategy);

ClientConfiguration configuration;

configuration.scheme = Http::Scheme::HTTPS;

configuration.connectTimeoutMs = 0;

configuration.requestTimeoutMs = 0;

configuration.retryStrategy = retryStrategy;

MyClient = new S3Client(creadentials, configuration);

BasicOperations();

// Instance of S3Client must be destroyed before call to ShutdownAPI function. Otherwise you'll get "Segmentation fault".

// This problem may occur when you create S3Client as local object, because it is destroyed when whole mian function is done.

delete MyClient;

//..................................................................................

ShutdownAPI(options);

return 0;

}

//----------------------------------------------------------------------------------------------------------------