Airflow scheduling system

Airflow scheduling is a web application that is completely written in python. It provides more functionality then Apache oozie.

To install airflow download from link given below :
http://airflow.readthedocs.org/en/latest/installation.html

 

 

Advertisements

How to generate graphs in python

Friends, Python is a language which is very easy & rich , having tons of library for various purpose. Today i am going to show you how to use matlplotlib library for ploting graphs as well as saving the graph in particular directory.

import numpy as np
import matplotlib.pyplot as plt
import os

N = 5
menMeans = (20, 35, 30, 35, 27)
menStd = (2, 3, 4, 1, 2)

ind = np.arange(N)  # the x locations for the groups
width = 0.35       # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(ind, menMeans, width, color=’r’, yerr=menStd)

womenMeans = (25, 32, 34, 20, 25)
womenStd = (3, 5, 2, 3, 3)
rects2 = ax.bar(ind + width, womenMeans, width, color=’y’, yerr=womenStd)

# add some text for labels, title and axes ticks
ax.set_ylabel(‘Scores’)
ax.set_title(‘Scores by group and gender’)
ax.set_xticks(ind + width)
ax.set_xticklabels((‘G1’, ‘G2’, ‘G3’, ‘G4’, ‘G5’))

ax.legend((rects1[0], rects2[0]), (‘Men’, ‘Women’))

def autolabel(rects):
# attach some text labels
for rect in rects:
height = rect.get_height()
ax.text(rect.get_x() + rect.get_width()/2., 1.05*height,
‘%d’ % int(height),
ha=’center’, va=’bottom’)

autolabel(rects1)
autolabel(rects2)

#plt.show()
def save(path, ext=’png’, close=True, verbose=True):
# Extract the directory and filename from the given path
directory = os.path.split(path)[0]
filename = “%s.%s” % (os.path.split(path)[1], ext)
if directory == ”:
directory = ‘.’
# If the directory does not exist, create it
if not os.path.exists(directory):
os.makedirs(directory)
# The final path to save to
savepath = os.path.join(directory, filename)
if verbose:
print(“Saving figure to ‘%s’…” % savepath),
# Actually save the figure
plt.savefig(savepath)
# Close it
if close:
plt.close()
if verbose:
print(“Done”)

save(“/path/to/file”, ext=”png”, close=False, verbose=True)

Python – java integration (Jython continue…)

Friends , Having knowledge of multiple language is good. but sometimes it become cumbersome to use the libraries written in one language into another. Jython provides a way to run python over JVM. Hence allows integration of both java & python. We can use java classes & function in python as well as python libraries in java simply.

Below i am going to give an example for such.

  • We are going to create interface in java which will be implemented in python. which are again getting called in java language
  1. Create a package name org.jython.book.interfaces & define the interace as given below.
     

    package org.jython.book.interfaces;// Java interface for a building object

    public interface BuildingType {

    public String getBuildingName();

    public String getBuildingAddress();

    public String getBuildingId();

    }

  2. Create a python module which implements the above interface.

from org.jython.book.interfaces import BuildingType

class Building(BuildingType):
def __init__(self, name, address, id):
self.name = name
self.address = address
self.id = id

def getBuildingName(self):
return self.name

def getBuildingAddress(self):
return self.address

def getBuildingId(self):
return self.id

3. Create another package called package org.jython.book.util

Create a  class named BuildingFactory.java

 

/*
* To change this license header, choose License Headers in Project Properties.
* To change this template file, choose Tools | Templates
* and open the template in the editor.
*/
package org.jython.book.util;
import org.jython.book.interfaces.BuildingType;
import org.python.core.PyObject;
import org.python.core.PyString;
import org.python.util.PythonInterpreter;

public class BuildingFactory {

private PyObject buildingClass;

/**
* Create a new PythonInterpreter object, then use it to execute some python
* code. In this case, we want to import the python module that we will
* coerce.
*
* Once the module is imported than we obtain a reference to it and assign
* the reference to a Java variable
*/
public BuildingFactory() {
PythonInterpreter interpreter = new PythonInterpreter();
interpreter.exec(“import sys\n” + “sys.path.append(‘/root/NetBeansProjects/JythonR/src/org/jython/book/interfaces/’)\n”+”from Building import Building”);
buildingClass = interpreter.get(“Building”);
}

/**
* The create method is responsible for performing the actual coercion of
* the referenced python module into Java bytecode
* @param name
* @param location
* @param id
* @return BuildingType
*/

public BuildingType create(String name, String location, String id) {
PyObject buildingObject = buildingClass.__call__(new PyString(name),
new PyString(location),
new PyString(id));

//buildingObject.__tojava__(Object.class);
BuildingType type = (BuildingType) buildingObject.__tojava__(BuildingType.class);
System.out.println(type.getClass());
return type;
}
}

4. Now simply write a main method

/*
* To change this license header, choose License Headers in Project Properties.
* To change this template file, choose Tools | Templates
* and open the template in the editor.
*/
package org.jython.book;

import org.jython.book.util.BuildingFactory;
import org.jython.book.interfaces.BuildingType;

public class Main {

private static void print(BuildingType building) {
System.out.println(“Building Info: ” +
building.getBuildingId() + ” ” +
building.getBuildingName() + ” ” +
building.getBuildingAddress());
}

/**
* Create three building objects by calling the create() method of
* the factory.
*/

public static void main(String[] args) {
BuildingFactory factory = new BuildingFactory();
print(factory.create(“BUILDING-A”, “100 WEST MAIN”, “1”));
print(factory.create(“BUILDING-B”, “110 WEST MAIN”, “2”));
print(factory.create(“BUILDING-C”, “120 WEST MAIN”, “3”));

}

}

 

BuildingFactory class will create a factory object which will convert python object into java object.

 

Run the program. Remember you will need to install jython on your system & need to add jython jar file into class library in order to run it!!.