Batch clipping multiple rasters with a single shapefile using arcpy - arcpy

import arcpy, glob, os
from arcpy import env
from arcpy.sa import *
# Set the input workspace
arcpy.env.workspace = r"F:\madhavi\images_to_clip"
arcpy.CheckOutExtension("Spatial")
# Absolute path to your mask layer
mask = "F:\madhavi\shapefile\shp_gang_only_final.shp"
# Copying all the input rasters in an array
rasters = arcpy.ListRasters()
# Loop through rasters, append names and save files
for raster in rasters:
output_raster = raster.replace(".TIF", "_clip.TIF")
rasterObject = arcpy.gp.ExtractByMask_sa(raster, mask, output_raster)
rasterObject.save(r"F:\madhavi\clipped_images")
There are two problems that I am encountering with this Python script and they are as follows:
Firstly, the script is clipping only the first raster in the folder: F:\madhavi\images_to_clip and not for all the rasters in that folder. Therefore, the for loop is not working properly.
By the way, the folder on which the for loop is working contains only two .TIF images.
Secondly, the script is not saving the outputs in the desired folder.
I am still getting "Parsing Error SyntaxError: EOL while scanning string literal (line 18)". Therefore, there is a problem in "output_raster = raster.replace(".TIF", "_clip.TIF")".

I think the problem is in this variable assignment:
mask = "F:\madhavi\shapefile\shp_gang_only_final.shp"
The error message is complaining about string literal -- so try instead ensuring it is a string literal:
mask = r"F:\madhavi\shapefile\shp_gang_only_final.shp"

Related

Running entire C++ pipeline as blackbox from R

I have a .cpp file (a class I think) that I consider as a blackbox: it just requires a .txt file in a specific format as input, and it writes another .txt file as output.
How can I use this code from within R? The Rcpp package seems like the way to go, but from browsing the web all the answers/posts I have seen deal with calling a single C++ function from R (e.g., with // [[Rcpp::export]]). But in my case, I'm not interested in a specific function. I just want to run the entire C++ script as a blackbox...
Independently of R, compile it with your C++ compiler
g++ imtd.cpp -o imtd
producing an executable and then create an R program to produce the input file, edgelist.txt, run the executable using the R shell command and read the output file edgelist-out.txt back in:
shell("imtd edgelist")
# read edgelist-out.txt into R
# - the first field of the first line contains the number of triangles
# - lines containing a comma have 3 fields separated by one or more punctuation characters
# - there are some class counts at the end which we recompute rather than read
L <- readLines("edgelist-out.txt")
no.of.triangles <- read.table(text = L, nrow = 1)[[1]]
# extract lines with a comma, replace punctuation with space & create 3 column data frame
DF <- read.table(text = gsub("[[:punct:]]", " ", grep(",", L, value = TRUE)))
# rather than read in the class counts compute them from DF
tab <- table(DF$V3) # table of class counts
You don't need Rcpp for this. The above allows treating the imtd.cpp file as a black box knowing only the formats of the input and output files, as requested.

clip analysis in arcpy

I have one shapefile that covers an entire city, and a list of shapefiles which are buffers in different places in the city. I want to clip the city with each buffer. I tried using ArcPy in Python but the code is not working. What am I doing wrong?
import arcpy
from arcpy import env
from arcpy.sa import *
env.workspace = "U:\Park and Residential Area\Test\SBA park_res_buffer_5\SBA.gdb"
infeature= "U:\Park and Residential Area\Test\park_res_merge.shp"
clipfeatture = arcpy.ListFeatureClasses("*", "polygon")
for i in clipfeatture:
outclipfeatture = arcpy.Clip_analysis(infeature,i)
outclipfeatture.save("U:\Park and Residential Area\Test\SBA park_res_buffer_5/"*i)
This is the appropriate syntax for using Clip in ArcPy:
arcpy.Clip_analysis(in_features, clip_features, out_feature_class)
so your for loop should instead be something like:
for i in clipfeatture:
outfeature = "U:\Foo\Bar\" + i
arcpy.Clip_analysis(infeature, i, outfeature)
I would also print() each file path string so you can check that its syntax is being used appropriately. Backslashes are escape characters in Python and can have special properties when followed by key letters.
I always put an r in front of any string that contains a file path, e.g. r"\\srvr\drv\proj\gdb.gdb\fc"; this tells Python it is a raw string and ignores the escape functions.
See link below for an entertaining analogy on handling backslashes in filenames.
https://pythonconquerstheuniverse.wordpress.com/2008/06/04/gotcha-%E2%80%94-backslashes-in-windows-filenames/
To do multiple clip with more than one clip features, you have to first create a list of all clip features and iterate them.
import arcpy
arcpy.env.workspace = file_path
fcList = arcpy.ListFeatureClasses()
for fc in fcList:
arcpy.Clip_analysis(input_feature, fc, output_feature)
Be sure to have different names for your multiple outpus. You can use arcpy.CreateUniqueName() to create distinct names such as Buffer.shp, Buffer_1.shp, Buffer_2.shp ...
You can also export the iterate feature selection tool from Model Builder if each place is unique.
http://desktop.arcgis.com/en/arcmap/10.3/tools/modelbuilder-toolbox/iterate-feature-selection.htm
# Import arcpy module
import arcpy
# Load required toolboxes
arcpy.ImportToolbox("Model Functions")
# Local variables:
Selected_Features = ""
Value = "1"
# Process: Iterate Feature Selection
arcpy.IterateFeatureSelection_mb("inputfeature", "fields", "false")

How to organize data from a .DXF file

I need to take apart a GDS file to do more detailed analysis on it.
In particular I need to count the numbers of design layers, the number of polylines in each layer and the vertices of each of the polyline for a design.
My trouble is creating class(s) with all that information. In the end I would like to be able to get the data for each vertex by specifying something like
Lyr[5].pl[3].vertx[7] Where 5 is the fifth layer, 3 is the third polyline in that layer and 7 is the seventh vertex on that polyline.
I can get the layers and polylines fine but can't get my mind around how to start adding the vertices.
BTW, I am not working directly with a GDS file but a format called .DXF which is
a simple single string per line text file.
Here is my code so far.
import sys #see sys modules using goggle
import string
import math
class layer:
def __init__(self,layer):
self.layer=layer
self.nplines=0
self.pl=[]
def add_pline(self,y):
self.pl.append(y)
def add_nplines(self):
self.nplines+=1
'''
class vertx
def __init__(self,n): ## Running out of ideas here
'''
def main():
my_file=sys.argv[1]
inFile=open(my_file,'r')
lyr=[]
nlayers=-1
## Get the layers
while 1:
s=inFile.readline()
if "CONTINUOUS" in s :
nlayers+=1
lyr.append(0)
s=inFile.readline() #burn one line in DXF file
s=inFile.readline() #name of the layer
lyr[nlayers]=layer(s) # save the layer
if 'POLYLINE' in s: break
inFile.close()
## Get the polylines
inFile=open(my_file,'r')
while 1:
s=inFile.readline()
if 'POLYLINE' in s:
s=inFile.readline() #burn a line
s=inFile.readline() #layer name
for i in range(0,nlayers+1):
if s==lyr[i].layer:
lyr[i].add_nplines()
if 'EOF' in s: break
inFile.close()
for i in range(0,nlayers+1):
print i,'Layer=',lyr[i].layer,' no plines= ',lyr[i].nplines
main()
You can use my ezdxf package to handle DXF files. It is available on PyPI https://pypi.python.org/pypi/ezdxf.
import ezdxf
dwg = ezdxf.readfile("your.dxf")
msp = dwg.modelspace() # contains all drawing entities
polylines = msp.query("POLYLINE") # get all polylines in modelspace
for polyline in polylines:
layer = polyline.dxf.layer # layername as string
points = polyline.points() # all vertices as (x, y [,z]) tuples
# for more see http://ezdxf.readthedocs.org
ezdxf handles all DXF versions and can also append data to existing DXF files or create new DXF files.

error in writing to a file

I have written a python script that calls unix sort using subprocess module. I am trying to sort a table based on two columns(2 and 6). Here is what I have done
sort_bt=open("sort_blast.txt",'w+')
sort_file_cmd="sort -k2,2 -k6,6n {0}".format(tab.name)
subprocess.call(sort_file_cmd,stdout=sort_bt,shell=True)
The output file however contains an incomplete line which produces an error when I parse the table but when I checked the entry in the input file given to sort the line looks perfect. I guess there is some problem when sort tries to write the result to the file specified but I am not sure how to solve it though.
The line looks like this in the input file
gi|191252805|ref|NM_001128633.1| Homo sapiens RIMS binding protein 3C (RIMBP3C), mRNA gnl|BL_ORD_ID|4614 gi|124487059|ref|NP_001074857.1| RIMS-binding protein 2 [Mus musculus] 103 2877 3176 846 941 1.0102e-07 138.0
In output file however only gi|19125 is printed. How do I solve this?
Any help will be appreciated.
Ram
Using subprocess to call an external sorting tool seems quite silly considering that python has a built in method for sorting items.
Looking at your sample data, it appears to be structured data, with a | delimiter. Here's how you could open that file, and iterate over the results in python in a sorted manner:
def custom_sorter(first, second):
""" A Custom Sort function which compares items
based on the value in the 2nd and 6th columns. """
# First, we break the line into a list
first_items, second_items = first.split(u'|'), second.split(u'|') # Split on the pipe character.
if len(first_items) >= 6 and len(second_items) >= 6:
# We have enough items to compare
if (first_items[1], first_items[5]) > (second_items[1], second_items[5]):
return 1
elif (first_items[1], first_items[5]) < (second_items[1], second_items[5]):
return -1
else: # They are the same
return 0 # Order doesn't matter then
else:
return 0
with open(src_file_path, 'r') as src_file:
data = src_file.read() # Read in the src file all at once. Hope the file isn't too big!
with open(dst_sorted_file_path, 'w+') as dst_sorted_file:
for line in sorted(data.splitlines(), cmp = custom_sorter): # Sort the data on the fly
dst_sorted_file.write(line) # Write the line to the dst_file.
FYI, this code may need some jiggling. I didn't test it too well.
What you see is probably the result of trying to write to the file from multiple processes simultaneously.
To emulate: sort -k2,2 -k6,6n ${tabname} > sort_blast.txt command in Python:
from subprocess import check_call
with open("sort_blast.txt",'wb') as output_file:
check_call("sort -k2,2 -k6,6n".split() + [tab.name], stdout=output_file)
You can write it in pure Python e.g., for a small input file:
def custom_key(line):
fields = line.split() # split line on any whitespace
return fields[1], float(fields[5]) # Python uses zero-based indexing
with open(tab.name) as input_file, open("sort_blast.txt", 'w') as output_file:
L = input_file.read().splitlines() # read from the input file
L.sort(key=custom_key) # sort it
output_file.write("\n".join(L)) # write to the output file
If you need to sort a file that does not fit in memory; see Sorting text file by using Python

Insert file (foo.txt) into open file (bar.txt) at caret position

What would be the best method, please, to insert file (foo.txt) into open file (bar.txt) at caret position?
It would be nice to have an open-file dialog to choose anything to be inserted.
The word processing equivalent would be "insert file" here.
Here is a substitute for foo.sublime-snippet, which can be linked to form files elsewhere:
import sublime, sublime_plugin
class InsertFileCommand(sublime_plugin.TextCommand):
def run(self, edit):
v = self.view
template = open('foo.txt').read()
print template
v.run_command("insert_snippet", {"contents": template})
From within a text command you can access the current view. You can get the cursor positions using self.view.sel(). I don't know how to do gui stuff in python, but you can do file selection using the quick panel (similar to FuzzyFileNav).
Here is my unofficial modification of https://github.com/mneuhaus/SublimeFileTemplates which permits me to insert-a-file-here using the quick panel. It works on an OSX operating system (running Mountain Lion).
The only disadvantage I see so far is the inability to translate a double-slash \\ in the form file correctly -- it gets inserted instead as just a single-slash \. In my LaTex form files, the double-slash \\ represents a line ending, or a new line if preceded by a ~. The workaround is to insert an extra slash at each occurrence in the actual form file (i.e., put three slashes, with the understanding that only two slashes will be inserted when running the plugin). The form files need to be LF endings and I'm using UTF-8 encoding -- CR endings are not translated properly. With a slight modification, it is also possible to have multiple form file directories and/or file types.
import sublime, sublime_plugin
import os
class InsertFileCommand(sublime_plugin.WindowCommand):
def run(self):
self.find_templates()
self.window.show_quick_panel(self.templates, self.template_selected)
def find_templates(self):
self.templates = []
self.template_paths = []
for root, dirnames, filenames in os.walk('/path_to_forms_directory'):
for filename in filenames:
if filename.endswith(".tex"): # extension of form files
self.template_paths.append(os.path.join(root, filename))
self.templates.append(os.path.basename(root) + ": " + os.path.splitext(filename)[0])
def template_selected(self, selected_index):
if selected_index != -1:
self.template_path = self.template_paths[selected_index]
print "\n" * 25
print "----------------------------------------------------------------------------------------\n"
print ("Inserting File: " + self.template_path + "\n")
print "----------------------------------------------------------------------------------------\n"
template = open(self.template_path).read()
print template
view = self.window.run_command("insert_snippet", {'contents': template})
sublime.status_message("Inserted File: %s" % self.template_path)

Resources