PySmartScriptersGuide

Important Concepts

PySMART data management revolves around two primary components - projects and records. A project is a collection of records and associated information. A record is an individual image or spectrum that can be manipulated as needed. In PySMART, both projects and records are objects that are used by the programmer.

Opening and Closing Projects

All records in PySMART are stored in projects. Projects must be opened, possibly saved, and closed to ensure that there are no problems with them. We first need to import the Project object:

from pysmart.project import Project

To create a new project (and overwrite any existing file), use

proj = Project(filename, newProject=True)

To open an existing project, use

proj = Project(filename)

If, after working on a project, you would like to save the changes, use

proj.save()

To close the project, use

proj.close()

. Be aware that not closing a project will cause Python to hang.

Optimized Batch Operations

Many of today's computers have processors with more than one core, each of which is able to run its own code. Thus, if we can use multiple cores simultaneously, we can vastly decrease the time it requires to process data. PySMART includes an easy to use batch job processor that automatically determines the number of processing cores available (on Linux and Windows) and processes data in parallel rather than serially. To be more specific, the batch processor creates a number of "threads" that are all able to execute simultaneously. While doubling the number of cores/threads will not result in a doubling of performance, it will result in a very noticeable gain.

The following is a code example that will run multiple tasks in parallel and add the results to a project:

from pysmart.utils import BatchJob
from pysmart.record import record

# the following line assumes you have created extraction requests that are stored in the variable "requests"
# you can also use this for any other function. The [] and {} should contain any arguments or keyword arguments
# to pass to the function. We also assume the current project is in the variable "project"
tasks = [(req.extract, [], {}) for req in requests]
job = BatchJob(tasks)
job.run()
project.startBatch()
errors = []
for spec in job.results:
    if isinstance(spec, Record): spec.addToProject()
    else: errors.append(spec)
project.flushBatch()
if errors: print errors

Example Script

The following scripts finds all FITS files in the current directory and extracts them using either a tapered column (low res) or full slit (high res). This does not use the batch job processing (but can easily be extended to do so).

from glob import glob
import os
from pysmart.pysmart import PySMART
from pysmart.project import Project
from pysmart.instruments.SpitzerIRS.image import SpitzerIRSImage
import pysmart.instruments.SpitzerIRS.extract as extract_irs

# find all the fits file in the current directory
files = glob('*.fits')

# create a new instance of PySMART
pysmart = PySMART()

# create a new project
project = Project('project.smp', newProject=True)

# convert all files into records
images = [SpitzerIRSImage(project=project, filename=filename) for filename in files]
print "Images Loaded"

# clean each record
images = [image.cleanImage() for image in images]
# add all the new images to the project
[image.addToProject() for image in images]
print "Images Cleaned"

# now, for each image, make an extraction request (full slit for high res,
# tapered column for low res)
requests = []
for image in images:
    if image.module[1] == 'L': requests.append(extract_irs.createAutoTaperedColumnRequest(project, image))
    else: requests.append(extract_irs.createFullSlitRequest(project, image))

# extract
spectra = [request.extract() for request in requests]
# add the spectra to the project
[spectrum.addToProject() for spectrum in spectra]
print "Images Extracted"

# save the project
project.save()

# for each of the spectra, save it to a file
os.mkdir('spectra')
[spectrum.file.writeto('spectra/%s'%(spectrum.filename,)) for spectrum in spectra]

# close the project
project.close()