Face Your Fears: Part 2 - An Amazonka Tutorial

Welcome Back

In the previous blog-post in this series, we discussed the importance of leveraging different cloud providers to deploy an effective DevOps strategy. However, with different providers come different APIs, each with its own conventions and data types, posing a risk of issuing inconsistent commands and increasing your exposure to downtime.

In the grander goal of designing a type safe orchestration and monitoring tool for LogoGrab's hybrid cloud, the last blog-post described how we took the first step by implementing a functional API to make requests to Scaleway.

In this blog-post we will see how we can talk to Amazon OpsWorks for retrieving our AWS based instances.

Fear of the Unknown

When I was exploring the idea of using Haskell as the language for writing the LogoGrab-Cloud tool, I came to an early point of fear.

I was so familiar with Python; throwing out scripts was easy knowing the ecosystem and the intricacies of the language. I had used boto numerous times to script out any AWS functionality we required. I then looked at amazonka, the Haskell equivalent. There was something so daunting about looking at the library, but at the same time, I had so much respect for how the library was made (the story can be found here).

To expel fear from others like me, I will break down how this library can be approached with more ease so that you can start hacking on Amazon services in Haskell today! The case I will go through in this post is the retrieval of EC2 and OpsWorks instances in order to convert them into a data type that contains information relevant to the LogoGrab-Cloud tool. Let's get started!

1. Installing Amazonka

The first thing to do is to get the libraries installed within your dependency manager; the one I used being stack. Amazonka is organized with two core dependencies amazonka and amazonka-core. The rest of the services are in their own packages, with the naming convention of amazonka-* where * represents the service.

To install the dependencies you can either run stack install amazonka, stack install amazonka-core, stack install amazonka-ec2 and stack install amazonka-opsworks directly, or if you are working within a project, you can add them to the .cabal file.

So now that all the dependencies are installed, we will start to look at how we can explore the library and figure out which parts we need. The first place we want to look at in order to get familiar with the code is the examples section of the git repository. Look at that: we already have an EC2 example! The second thing we will need to have open is the EC2 section of the library itself. We will use the EC2 example to write our OpsWorks code.

2. Getting the Environment (Credentials Discovery)

Now we will look at how to initialize the credentials. A common way of doing this is to allow the discovery of the credentials via environmental variables or an AWS configuration file. This can be done as per the example code:

env <- newEnv Discover  

But what if you want to provide the access and secret key yourself, just as I did? Well, the first thing to check is where newEnv is coming from and which arguments you can provide. Hopping over into the github repo we can search for newEnv and see where it's defined. We can see that it gets exported from Control.Monad.Trans.AWS. Let's hop into our ghci REPL and run the following:

> import Control.Monad.Trans.AWS
Control.Monad.Trans.AWS> :browse Control.Monad.Trans.AWS  
...

newEnv ::  
  (Applicative m, Control.Monad.IO.Class.MonadIO m,
   exceptions-0.8.3:Control.Monad.Catch.MonadCatch m) =>
  Credentials -> m Env

...

As we can see, newEnv takes some type Credentials and produces an m Env with some type constraints on m. The next question is, how do we construct Credentials? Well there's another useful command we can run in the REPL:

Control.Monad.Trans.AWS> import Network.AWS.Auth  
Network.AWS.Auth Control.Monad.Trans.AWS> :i Credentials  
data Credentials  
  = FromKeys AccessKey SecretKey
  | FromSession AccessKey SecretKey SessionToken
  | FromEnv Text Text (Maybe Text) (Maybe Text)
  | FromProfile Text
  | FromFile Text FilePath
  | Discover
      -- Defined in ‘Network.AWS.Auth’
instance Eq Credentials -- Defined in ‘Network.AWS.Auth’  
instance Show Credentials -- Defined in ‘Network.AWS.Auth’  

So now we know how Credentials can be constructed and that they are defined in Network.AWS.Auth; all very useful information! The plus side is that we are already getting used to understanding how the project is structured and how we can find the functionality we want.

So now our main should look something like the following:

main = do  
  env <- newEnv $ FromKeys (AccessKey "access-key") (SecretKey "secret-key")

3. Describe Instances

The next step is figuring out how we can make a call to an AWS service. Looking back at the example code we can see:

runResourceT . runAWST env . within r $  
        paginate describeInstances
        ...

For this part we will look at the code that was used in my case and break down the important parts to figure out what's going on:

import qualified Network.AWS.OpsWorks as OpsWorks

main = do  
  env <- newEnv $ FromKeys (AccessKey "access-key") (SecretKey "secret-key")instances <- runResourceT . 
  logoGrabInstances <- runAWST env . within NorthVirginia $ do
        instances <- send $ OpsWorks.describeInstances & OpsWorks.diStackId ?~ "stack-id"

Let's check each function that is being used here, starting with runResourceT:

Network.AWS.Auth Control.Monad.Trans.AWS> :t runResourceT  
runResourceT  
  :: monad-control-1.0.1.0:Control.Monad.Trans.Control.MonadBaseControl
       IO m =>
     resourcet-1.1.9:Control.Monad.Trans.Resource.Internal.ResourceT m a
     -> m a

-- Removing the package names
runResourceT  
  :: MonadBaseControl IO m => ResourceT m a -> m a

This might be a bit confusing (at least it was for me), but let's work it out. MonadBaseControl is a general way of going down to some base monad, doing some work in that context, then re-entering our monad stack. In this case, our base monad is IO. So given some ResourceT m a we run some computations and output our MonadBaseControl.

Let's continue to explore what these functions are doing by checking the types of our next step of composition:

-- make a fake Env to explore the types
Network.AWS.Auth Control.Monad.Trans.AWS> let env = undefined :: Env

Network.AWS.Auth Control.Monad.Trans.AWS> :t runResourceT . runAWST env  
runResourceT . runAWST env  
  :: MonadBaseControl IO m => AWST' Env (ResourceT m) a
  -> m a

So now we need to provide a type that has a MonadBaseControl IO constraint on the type AWST' Env (ResourceT m) a and this will give us back our m a.

The final composition is the function within so let's check the type of this function:

within  
  :: (HasEnv r, MonadReader r m) =>
     Region -> m a -> m a

This is saying that within wants a Region followed by a MonadReader stack with the constraint of HasEnv on the reader environment r. We can see that AWST' is part of a MonadReader stack if we inspect it in the REPL:

Network.AWS.Auth Control.Monad.Trans.AWS> :i AWST'  
type role AWST' representational representational nominal  
newtype AWST' r (m :: * -> *) a  
  = Control.Monad.Trans.AWS.AWST' {unAWST :: ReaderT                                                         r m a}
      -- Defined in ‘Control.Monad.Trans.AWS’

So combining all that together we get:

:t runResourceT . runAWST env . within NorthVirginia
runResourceT . runAWST env . within NorthVirginia  
  :: MonadBaseControl IO m 
  => AWST' Env (ResourceT m) a -> m a

To run this, we need to provide something with the type AWST' Env (ResourceT m) a.

Now that we've covered the function that executes our inner do block, let's look at what we are doing inside there. The first line looks like this:

instances <- send $ OpsWorks.describeInstances & OpsWorks.diStackId ?~ "stack-id"  

Once again, we'll repeat the process of inspecting the types of our functions, the first up being send:

Network.AWS.Auth Control.Monad.Trans.AWS> :t send  
send  
  :: (AWSRequest a, HasEnv r, MonadReader r m, MonadResource m, MonadCatch m) 
  => a -> m (Rs a)

We have seen a few of these type constraints already, so let's point out some of the new ones. AWSRequest is a typeclass that represents any type of request we can make to AWS (this one being DescribeInstances). We can confirm this by running :i OpsWorks.DescribeInstances. MonadResource is the typeclass that says our monad stack should have an instance of ResourceT within it. I will leave MonadCatch as an exercise to the reader.

The Rs a in the type signature is some type-family magic which I am not informed enough to comment on as of yet. It can be summarized as: if I ask for a DescribeInstances I will get back a DescribeInstancesResponse.

So send takes some AWS request and gives us an AWS response. Well then, what's the rest of this code about?

OpsWorks.describeInstances & OpsWorks.diStackId ?~ "stack-id"  

We need to tell the request how to get the OpsWorks instances we are interested in and, in this case, we will provide the Stack ID. The operators & and ?~ are lens operators exported from Control.Lens. & is function application in reverse a -> (a -> b) -> b. ?~ is a way of setting an optional value. We can find what Opsworks.diStackId is here, since it's of type Lens' DescribeInstances (Maybe Text) . The field in DescribeInstances is an optional field as we can retrieve instances in multiple ways. Thus, we use the ?~ to set the optional field of our Lens'. We then use the function application & along with OpsWorks.describeInstances to get our final OpsWorks.DescribeInstances.

4. Converting Instances

So we have our instances in the form of a DescribeInstancesResponse, but what we really want is the instances in a list. If we look back at the documentation, we can find dirInstances which again is Lens' that can give us back [Instance].

So we can extend our code to the following:

instances <- send $ OpsWorks.describeInstances & OpsWorks.diStackId ?~ "stack-id"  
let instanceList = view OpsWorks.dirsInstances instances  

Now that we have the instance list, we want to convert the OpsWorks instance into our custom data type. For this, we will keep it simple:

data LogoGrabInstance = LogoGrabInstance {  
    name :: Text
  , publicIP :: Maybe Text
} deriving (Show)

toLogoGrabInstance :: OpsWork.Instance -> Maybe LogoGrabInstance  
toLogoGrabInstance ins =  
  let name = ins ^. OpsWorks.iHostname
      publicIp = ins ^. OpsWorks.iPublicIP
  in fmap (\hostname -> LogoGrabInstance hostname publicIp) name

The things to note here are that the functions i* are lenses for the Instance type, which can be found here, and ^. is the same as view for getting the value of a lens.

Our final code example now looks like this:

main = do  
  env <- newEnv $ FromKeys (AccessKey "access-key") (SecretKey "secret-key")
  logoGrabInstances <- runResourceT . runAWST env . within NorthVirginia $ do
      instances <- send $ OpsWorks.describeInstances & OpsWorks.diStackId ?~ "stack-id"
      let instanceList = view OpsWorks.dirsInstances instances
      return $ map tologoGrabInstance instanceList
  forM_ logoGrabInstances (liftIO . print) -- print out our instances


Lessons Learned

Going through this library taught me an important lesson about programming and learning: do not be afraid of anything. If it feels like it isn't within your capability now, then spend some time to really look at the problem. You will find an answer and you will come out the other side happier. I have also seen some people fetishize the "difficulty" of Haskell. It is not any more difficult than learning how to code for the first time (which I personally struggled with). I wasn't a Python master on the first day of the job either, but rather, it took time to learn the ins and outs of the language. With enough practice and time, Haskell is as easy as any other language available to us, but allows us to make less mistakes, which is a big win for me!