Task Execution through Mail

This is one of the most interesting thing that I have worked on and implemented in one of my project. One can actually perform various actions based on free text message sent over mail. My use case was to approve the ticket through mail in ticketing tool. Let’s see how we can implement this.

Mailing Address

The most basic requirement is mailing address to which we will send and receive the messages. Now all the messages/mails coming to that email address needs to be redirected to the Mail Parser (application running on server to read mail text). Redirection can be done by configuring mail forwarding for that mailbox in mail exchange (https://technet.microsoft.com/en-us/library/dd351134(v=exchg.160).aspx).

Message Receiver and Transporter

This is the most trickiest part of whole setup. Setting up mail receiver includes lot of configuration.

  • Sendmail: Sendmail service required to be running to receive the forwarded messages coming to the mailbox. It is the default mailing transfer agent on the RedHat Linux distribution.

/etc/init.d/sendmail start

  • Change default Sendmail to allow sendmail to receive mail: By default sendmail receives mail from local server, but as we need to accept mails from external mail exchange so change the described setting to allow mails from other sources.

Edit the file /etc/mail/sendmail.mc and change/comment the line:

From: DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')
To:   dnl DAEMON_OPTIONS(`Port=smtp,Addr=127.0.0.1, Name=MTA')

  • Relay: We have allowed mail receiving from all sources. But it is a good practice to define the legit senders and relay mails received by them only. Or in other sense block the spammers.

File: /etc/mail/access

Connect:localhost.localdomain           RELAY
Connect:localhost                       RELAY
Connect:127.0.0.1                       RELAY
Connect:server1.mailexchange            RELAY
Connect:server2.mailexchange            RELAY

  • Alternate host names: Sendmail will not accept mail for a domain unless it is permitted to do so. It is necessary to define all the alternate host names of the server.

File: /etc/mail/local-host-names

server1.mailexchange
mail1.exchange

  • Local mail user: All the mails coming to particular domain on sendmail are sent to particular user which must be there on server. So create local user which will receive all mails for particular domain and process it.

Command: useradd sample

  • Separate emails by domain: It is optional but will be required if you have multiple domains and want to segregate mails coming from all domains. Map each domain to particular user to which mails will go.

File: /etc/mail/virtusertable

sample@domain1.com        sample
sample@domain2.com        test
test@domain2.com          test

  • Generate a new configuration file: Post all the changes run below command to load new config.

m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

  • Restart sendmail:

/etc/rc.d/init.d/sendmail restart

Mail Parser

This is one another important part of setup. Till now sendmail has sent the mail to specific user of domain. Now we need to process the mail and filter out spammers by matching the valid formats. There are few options available but the one I used is procmail (mail delivery agent).

Create file .procmailrc inside home directory of local mail user and define configuration to filter the message and pass it to application to parse the mail and perform the task.

/home/sample/.procmailrc

#.procmailrc
LOGFILE=/home/sample/procmail_log
VERBOSE = YES
:0
* ^(to|Cc|To|cc).*sample@domain1.com
| /usr/bin/python2.7 /opt/scripts/email_approval.py

procmail is automatically invoked by sendmail as soon mail comes. The message has reached to your application, now it is all up to you to parse it and perform the action based on mail text message.

Multi-master Puppet with shared CA

Do we really need multiple master setup ?

How do I know if I need more than one Puppet master ?

Should I move from open source to Puppet enterprise as my infrastructure is increasing ?

These are some of the common questions that comes in mind while setting up/ maintaining the Puppet to manage large infrastructure. In this blog we will not talk more about how to configure multi master setup, that you can easily get from Puppet’s official documentation, but will focus on how to make the whole system scalable, highly available and its reporting using open source Puppet.

Why do we need multi-master setup ?

Well, with the increase in infrastructure, Puppet also needs to grow with it!!

  1. Ability to scale as needed quickly as more systems are deployed with Puppet. We can support the infrastructure with single master up to certain limit. What exactly this limit is:
    • Number of resources declared in manifests.
    • Number of nodes.
    • Time to compile catalog.
    • Time between 2 successive run.
  2. Distribution of agent load.
  3. Flexibility to test and roll through upgrades.
  4. To be prepare for disaster recovery.

Architecture of highly available Multi-master Puppet setup

architecture

Above architecture is designed in a way to handle the initial setup as well as scaling of system without any user login required on Puppet servers.

It is always a best practice to keep the code in version control system so that multiple people can work on it and one can see latest code from there itself in case required, rather than accessing it through Puppet server. Once we push the changes, we can have build automation system like Jenkins to push the code to specific Region servers with their respective files if they are different.

Manage Puppet setup through Jenkins

As I mentioned earlier that no user login is required to manage Puppet setup, make the Puppet servers password less through Jenkins machine and define all the steps required to setup a Master/CA server in Jenkins job for initial setup as well as scaling it out.

Now just imagine how easy it is to add one additional Puppet master or setup complete replica of system for new region by passing just an IP address to Jenkins job.

Code Integrity

As we are talking of multi-master setup here, hence it becomes very important to maintain the integrity of code/modules across all Puppet masters. To make it simple we can keep the Puppet repository in single CA server and share the same directory  with all Puppet masters. This way Jenkins will push the new changes to single CA server and same code will automatically be available to all puppet masters.

Load Balancer/ DNS

Now the question arises how do production nodes connect/ request to multiple Puppet masters. There are multiple ways of handling this, either by Round-Robin DNS method or Load Balancer method. I feel the latter to be a better method as it is very easy to distribute the load among various Puppet masters and can also help in disabling/ enabling the master whenever required. All the Puppet clients will contain single FQDN as server which is mapped to Load Balancer VIP in DNS, and that contains pool having all the Puppet masters as members. In case of multiple regions, DNS will resolve to its respective VIP record.

NOTE: All the Puppet clients will communicate to Puppet CA server just for initial certificate registration and later for authentication of it, rest complete load is handled by Puppet masters (Non CA servers).

Configuring an additional Puppet Master

This is very important part of complete discussion, how to setup Puppet master in case of multi-master Puppet.

  • In bootstrap.cfg, disable ca authority.
  • Copy ssl/ca/ca_crl.pem file from CA server.
  • Configure dns_alt_names in the [main] block of puppet.conf.
  • Destroy local certificates.

sudo rm -r $(puppet master --configprint ssldir)
puppet cert clean master-A.region.com          

  • Generate special certificates on Puppet master using dns_alt_names.
  • Rerun Puppet and master configurations will sync down.

Disaster Recovery

What if one Puppet master goes down ?

This is the main power of multi-master Puppet. It will automatically handle the failure of one master provided the capability of rest of the masters to manage enough load. Load Balancer will automatically disable the failed node, hence all the requests will be directed to the remaining masters. Our cluster will keep on running.

What if all Puppet masters go down ?

Yes it will be a Puppet down time and is a bit problematic situation but not a serious one, as it has no direct impact on Puppet clients. As soon as the Puppet masters comes up, the system is back to normal state.

Even in case of machine failure, we have Jenkins automation jobs (another master piece of architecture) in place to configure the setup on new machines in minutes, which we used to do the whole new setup in starting.

What if CA server goes down ?

It is advisable to keep one machine as CA backup server, pre-install Puppet configuration on it and setup scheduler to sync complete directory structure containing certificates. This is kind of master slave setup. As soon as main machine goes down, change the DNS record to point to backup CA server.

You must be thinking that what will happen if we do not have CA backup server or the backup is unavailable/ corrupted. Obviously this is an anxious situation as one will have to delete and regenerate certificates on all Puppet clients, but I have very easy way of handling it via MCollective(An orchestration framework comes along with Puppet). You must have seen MCollective service while installation of Puppet. It is a push mechanism unlike Puppet a pull mechanism. So through MCollective client you have full control over complete production and with just single command you can recover whole infra back.

It is always a best practice to use FQDN for each server entry rather than direct IP address, it helps us in case of failure scenarios just by flipping IP in DNS as explained above. ca_server=puppetca.region.com

What if Puppet DB goes down ?

Similar to CA backup server, configure master-slave setup of Puppet DB. Along with master-slave setup, it is always a good practice to schedule daily backups to some central location which helps us recovering the data in case of disaster.

Puppet Reporting

There is one open source reporting tool available “Puppet Explorer” by Spotify. I have personally used it, so can definitely recommend it to all. It shows interesting required information like total nodes, unresponsive nodes, last failed report, shows events defining resources executed on specific node and also provides search bar to filter any node information. You can get it from Puppet Forge too.

puppet module install spotify-puppetexplorer

This is all about my experience with Puppet. Feel free to post comments for any queries or feedback.

Automatic Deployments: Celery Scheduler

Celery is a Pythonic way of executing tasks/jobs asynchronously on distributed message passing. We can schedule or execute tasks immediately in a concurrent manner on a single or more worker servers using multiprocessing.

In my last blog Continuous Delivery I discussed about implementation of continuous delivery model in an organization. And now we will move one step ahead to automate the execution of deployments using Celery.

There was a time when “Release Team” used to be busy in doing deployments 24×7 on thousands of servers, and now is the time where we do not need any such team. Development team can rollout their changes just by scheduling it and the deployment tool described in the below architecture will take care of everything.

architecture

Celery has one very useful feature called “Celery Canvas”. It has the ability to combine multiple tasks to create workflows. We can define the dependencies among various tasks and execute them based on each other’s output. Different options in Canvas are:

  1. Group: It is a signature that takes a list of tasks that should be applied in parallel.
  2. Chain: It lets us link together signatures so that one is called after the other, essentially forming a chain of callbacks.
  3. Chord: It consists of a header group and a body, where the body is a task that should execute after all of the tasks in the header are complete. In short combination of both group and chain.

Whatever is the release plan, we can easily convert it into Canvas form. Through this we can make independent tasks as part of group to execute them in parallel and save time.

A very good use-case where I used it – to execute the release in multiple data centers concurrently. It is so simple that you just need to pass the procedure in group() and Celery will take care of the rest. You will not jave to face any of the crappiness that comes with multi threading.

Celery is scalable vertically as well as horizontally. So systems configuration will never be bottleneck for you. With Celery Canvas we defined the release workflow and achieved some level of parallelism and with MCollective we got one more level of parallelism with its inbuilt capabilities.

The Marionette Collective (MCollective) is a server orchestration/parallel job execution framework available from Puppet Labs (http://docs.puppetlabs.com/mcollective/). It can be used to programmatically execute administrative tasks on clusters of servers. MCollective along with its middleware system acts as pub-sub model where we publish task to be done and target servers, and those servers will execute the task and respond back the status code.

Summarizing The Architecture:-

  1. As soon as the developer commits the code to VCS, our build automation server which keeps on polling the VCS gets to know about new change and it starts building the code, running various test suites and then upload the deliverable to central artifact repository.
  2. Then QA raises the request to send change to production with few key inputs like – deliverable to roll-out, target servers, scheduled time.
  3. As the request comes to deployment tool, it schedules the respective project release in Celery.
  4. Celery beat keeps on polling the database within the defined interval to check if there is any task/release to execute.
  5. When the time comes, scheduler sends the task/release to Django app where we get all configuration/ release plan (already configured in database for all projects) from DB and make the workflow in form of Celery canvas.
  6. Celery then assigns the task to various available workers one by one via RabbitMQ broker.
  7. Now these workers logically communicate to MCollective Client which passes the messages to ActiveMQ broker to which all the Mcollective servers(agent present on all production servers) are connected.
  8. The targeted servers pick the message and executes it locally on servers.

This way we have reached to the stage of automatic deployments. And the best part of architecture is, we implemented the same with all open-source tools along with in-house deployment tool written in Django.

Continuous Delivery

Continuous Delivery

Continuous Integration, Continuous Deployment, or Continuous Delivery…. These are the terms which you all must have heard, but still can’t differentiate between them. These are the practices which all the organisations want to follow and but have not achieved yet may due to lack of processes, tools or resources. Below picture will explain you everything.

What is Continuous Delivery ?

Continuous delivery does not mean that you are deploying every commit/ change to production every time which, it is implementing such a process to get every change proven to be deployable to production. And Continuous integration includes code commit to build automation to unit tests.

But this is not possible for all organisations. There might be the requirement of approvals in between to rollout the changes to production. Also there could be a separate team to just deploy changes and in that case you could only implement continuous integration and deployment process would be separate. In this blog I want to focus more on deployment automation area.

Why to implement Continuous Delivery ?

  1. The most important speed, to get faster delivery
  2. Better quality, as your code gets verified on every commit.
  3. Capacity and scaling. Bigger teams can work together flawlessly.
  4. Reliability, as human intervention is very less.

What are the requirements for Continuous Delivery ?

  1. Source code
  2. Build Automation
  3. QA Automation
  4. Staging environment
  5. Ticketing tool for approvals
  6. Deployment
  7. Monitoring

How to achieve this with open source tools?

  1. First thing is source code for which I would suggest GIT (http://git-scm.com/about). We have many open source tools avaliable for hosting git repositories like Gitlab (https://about.gitlab.com/features/ ), Gerrit etc.Gitlab provides you very clean UI with some cool features like user management, code review, web hooks, code snippet etc.
  2. Second thing is Continuous Integration for which we have 2 good open source tools available:

    Here you can integrate it with you repository and can create various pipelines and schedule them one after other. On every commit build will get start automatically and next pipeline will be triggered only if first one gets successful. So from here anyone could track their build results. Also integrate various test automation pipelines here. Once it passes the QA automation then add a pipeline to deploy it to staging environment.

    This way you can achieve Continuous Integration. For better understanding look at the below picture.

  3. Now the third thing is ticketing tool. It is very difficult to track thousand of emails for approvals of release. Better implement some tool for it and also integrate it with CI tool for latest builds and with deployment tool.For this I found very unique tool named Activiti (http://activiti.org/).Here you just need to make flowcharts same as you used to make in your childhood days 🙂 You can make different processes with its approval workflow. It’s activecommunity and provides you API to integrate it with some other framework.I used it’s API and integrated it with python framework Django.Below is the sample that I have created showing various stages.
  4. Now another thing is Deployment automation which is the most tricky part. In most organisations it is the manual task which ends up into errors. This isthe area where most people look for some automation to smoothen deployment process.We have various options here:
    1. Rundeck (It uses SSH mechanism)
    2. Chef
    3. Puppet collaborated with MCollective (http://docs.puppetlabs.com/mcollective/)Here I will focus on MCollective. It is a framework to build server orchestration or parallel job execution systems. Let’s go bit deeper into this and understand what it is.
      • It can interact with small to very large clusters of servers
      • It comes with simple to use command line tools to call remote agents.
      • You can also write your own agents
      • It allows you to write simple RPC style agents, clients and Web UIs in an easy to understand language – Ruby
      • It needs a publish/subscribe middleware system of some kind for all communications which are ActiveMQ and RabbitMQ.

      With this we can deploy on multiple servers together which makes it fast. Please don’t forget to configure messaging queue with proper settings like queue size, number of queues, throughput, min/max memory.

  5. Now comes the monitoring. For this you can either create separate pipeline to run post deployment tests or you can also integrate it with loadbalancer tocheck its monitoring status.

This is all about Continuous Integration and Deployment Automation. Please feel free to comment for any suggestions or help.

pip vs easy-install

Every Python developer must have used pip and easy_install to install Python modules. But it still remains a confusion “What’s the difference, anyhow? “. This blog post is to give you clear understanding of which one to use.

Key Terms:

First of all I would like to explain few terms for Python beginners.

  • pip – pip is a tool for installing Python packages from the Python Package Index. Starting with Python 3.4, it is included by default with the Python binary installers.
  • easy_install – It is a package manager for Python that provides a standard format for distributing Python programs and libraries (based on the Python Eggs format).
  • Python Package Index (PyPI) – The official third-party software repository for the Python programming language.
  • Setuptools: Setuptools is a package development library designed to facilitate packaging Python projects by enhancing the Python standard library distutils (distribution utilities).

How to install a Python module ?

  • The most basic way to install a package is to download the source code and run ‘python setup.py install’ inside the package directory.

#wget https://pypi.python.org/packages/source/m/mechanize/mechanize-0.2.5.tar.gz#md5=32657f139fc2fb75bcf193b63b8c60b2
#tar -xvzf mechanize-0.2.5.tar.gz
#cd mechanize-0.2.5
#python setup.py install

  • Second way to install module is using easy_install (default Python package manager).

#easy_install mechanize
Searching for mechanize
Reading https://pypi.python.org/simple/mechanize/
Best match: mechanize 0.2.5
Downloading https://pypi.python.org/packages/source/m/mechanize/mechanize-0.2.5.zip#md5=a497ad4e875f7506ffcf8ad3ada4c2fc
Processing mechanize-0.2.5.zip
Writing /tmp/easy_install-FiMleV/mechanize-0.2.5/setup.cfg
Running mechanize-0.2.5/setup.py -q bdist_egg --dist-dir /tmp/easy_install-FiMleV/mechanize-0.2.5/egg-dist-tmp-SKWpzt
Adding mechanize 0.2.5 to easy-install.pth file
Installed /usr/local/lib/python2.7/site-packages/mechanize-0.2.5-py2.7.egg
Processing dependencies for mechanize
Finished processing dependencies for mechanize

  • Third way to install module is using pip

#pip install mechanize
Downloading/unpacking mechanize
Downloading mechanize-0.2.5.tar.gz (383kB): 383kB downloaded
Running setup.py egg_info for package mechanize
Installing collected packages: mechanize
Running setup.py install for mechanize
Successfully installed mechanize
Cleaning up...

How pip is better than easy_install ?

Now as a Python developer I would recommend pip over easy_install. To illustrate, I have list down some points which shows how pip is better than easy_install.

  1. Partial-completed installation:Pip downloads source code and builds it whereas easy_install downloads binaries. This is most crucial point which signifies that partially-completed installation doesn’t occur in case of pip.

    Easy_install doesn’t do a whole lot of up-front checking to make sure it’ll actually be able to install both the requested package and the dependencies, and this can lead to problems if something in the dependency chain ends up uninstallable.

    Meanwhile, pip looks before it leaps, can bail out early if it’s not going to be able to install your package and will leave behind a useful log file explaining what went wrong.

  2. Pip requirements file: This is really the killer feature pip has, because it means you can set up a simple plain text file specifying a list of packages (and just as important, which versions of which packages), and get pip to install them all and your project is ready to be used on different machine within seconds.To see how handy this can be, let’s take a simple example:

    requirements.txt

    BeautifulSoup==3.2.0
    Django==1.3
    Fabric==1.2.0
    Jinja2==2.5.5
    South==0.7.3

    #pip install requirements.txt

  3. Integration with virtualenv: Naturally pip integrates quite nicely with virtualenv. Normally, when working in an active virtualenv, Python packaging/installation tools (pip included) will install into that virtualenv, but pip also lets you specify a virtualenv to install into (using the -E flag), and ceate a new virtualenv and install into it.
    Let’s take a simple example: suppose you have an application which runs fine against the current Django release (1.0.2), and you want to play around a bit with the Django development trunk to try out a new feature. You could create a requirements file specifying docutils (so the automatic Django documentation will work) and the Subversion URL for Django trunk:

    docutils==0.5
    -e svn+http://code.djangoproject.com/svn/django/trunk#egg=Django

    Save this into a file named, say, django-requirements.txt. Then (assuming you have pip and virtualenv installed), run the following:

    pip install -E django-trunk -r django-requirements.txt
    source django-trunk/bin/activate

How easy_install is better than pip ?

Only one positive thing which I found in easy_install over pip is in windows environment. A number of packages on PyPI have binary eggs for Windows uploaded. For example, on Windows easy_install lxml will work fine and install precompiled binaries, whereas pip install lxml will try to build from source and almost certainly fail unless the user has gone to a great deal of trouble to setup a Visual Studio environment and have a working build of libxml2.